Data enrichment and Batch processing Cloud -> CDC Kafka -> Raw -> dbt-like transform -> HDFS Hive and DataVault 2.0 Postgres DDS -> Data Marts, OneBigTable Focused on building a set of inner web services for analysts. Being at product-company, I was able to work with a bunch of different tools, design architecture and build some tangible solutions for our clients (b2b). My internal tool for the company outbranched into a successful project which gives 10% of the revenue. Day at work: ✦ 3hrs ➥ Fresh data is being processed on daily basis. I have to build and monitor ETL pipelines, extract data from clouds into our DWH / DataLake. Data Engineer responsibilities. ✦ 2hrs ➥ POC for new/existing clients, hypothesis, provide API. ✦ 3hrs ➥ Develop and maintain a set of inner services next to DWH, including custom solutions for programmers, analysts and assessors, helpers DataVault 2.0 -> Data Marts. Main Stack: ✓ DataEngineer: HDFS, Kafka, Hive Metastore, parquet, Airflow, MPP, Pyspark, Spark Streaming ✓ Python, mypy, pytest ✓ wsgi: Django, DRF, Celery ✓ asgi: FastAPI, pydantic, aiohttp, asyncio, httpx ✓ Golang: pgx, clickhouse-go ✓ AWS RDS, EC2, Lambda, Amazon S3 ✓ Bash, Linux, Docker, docker-compose, On-premise, basic k8s skills ✓ SQL (orm and raw) PostgreSQL, NoSQL DynamoDB, Redis, OLAP OLTP, OneBigTable Clickhouse ✓ Prometheus, ELK ✓ ci/cd git pull-requests tests Sometimes: ✓ GPU OpenGL ES compute shaders for neighboring team ✓ fullstⒶck, reⒶct for our API, ✓ d3.js svg custom charts, Superset

Работа со стеком Hadoop, администрирование кластера, Web-сервисов, map-reduce пайплайны Подготовка витрин данных для своей команды, интеграция с Data Platform Подробности по требованию

ВМК, Прикладная математика и информатика

Mail.ru, BigData, DataMining. Близнец ШАД. id=alumni/1/139/