📚 Curiosity Chronicles

AWS

Properties2
description02 AWS
tagsinterview, aws, cloud

9 min read

AWS

You have 4 AWS certs (SAA, DVA, DAS, DEA). Expect “how does it work internally?” and “where have you used it?” — not “what is S3?”.

IAM

1. What is IAM?

IAM (Identity and Access Management) securely controls who can authenticate and what AWS resources they can access.

2. IAM User vs IAM Role

IAM User — permanent identity with long-term credentials.

IAM Role — temporary identity assumed by users, services, or accounts. Preferred for applications.

3. IAM Policy

A JSON document defining permissions (Allow/Deny) for AWS resources and actions.

4. Identity-based vs Resource-based Policies

Identity-based — attached to users, groups, or roles.

Resource-based — attached directly to resources (e.g., S3 bucket policy, Lambda resource policy).

5. What is STS?

AWS Security Token Service provides temporary credentials by assuming IAM roles via AssumeRole.

6. Why use Roles instead of Access Keys?

Roles provide temporary credentials, improve security, and eliminate long-lived secrets that can be leaked.

7. Cross-Account Access

Use IAM Roles with trust policies and STS AssumeRole to securely access resources across AWS accounts.

8. Principle of Least Privilege

Grant only the minimum permissions required to perform a task.

9. Permission Boundary

Defines the maximum permissions an IAM entity can have, regardless of what policies are attached.

10. Trust Policy

Specifies who can assume a role (the principal). Distinct from permission policies, which specify what actions are allowed.

Amazon S3

11. S3 Storage Classes

ClassUse case
StandardFrequent access
Intelligent-TieringUnknown/changing access patterns
Standard-IAInfrequent access, retrieval fee
One Zone-IAInfrequent + single AZ
Glacier InstantArchive, ms retrieval
Glacier FlexibleArchive, minutes-hours retrieval
Glacier Deep ArchiveLowest cost, 12hr retrieval

12. Glacier Restore

Data stored in Flexible or Deep Archive cannot be accessed immediately. It must be temporarily restored to S3 Standard before reading. Restoration time depends on the retrieval tier (Expedited, Standard, Bulk) and can take from minutes to hours.

13. Versioning

Maintains multiple versions of an object to protect against accidental deletion or overwrites.

14. Lifecycle Rules

Automatically transition or delete objects based on age or storage class rules.

15. Cross-Region Replication (CRR)

Replicates objects automatically to another AWS Region for disaster recovery or latency reduction.

16. Multipart Upload

Uploads large files in parallel by splitting into parts — better performance and resiliency for files > 100 MB.

17. Pre-Signed URL

Temporary URL allowing secure access to private S3 objects without exposing credentials. Time-limited.

18. Server-Side Encryption

  • SSE-S3 — S3 manages keys
  • SSE-KMS — KMS manages keys (audit trail via CloudTrail)
  • SSE-C — customer provides keys
  • Client-side — encrypt before upload

19. Event Notifications

Triggers Lambda, SQS, or SNS when objects are created, deleted, or modified. Also via EventBridge for richer routing.

20. Object Lock

Prevents object deletion or modification for compliance or legal hold. Write Once Read Many (WORM).

21. S3 Performance Tips

  • Use date-based partitioning to organize data logically
  • Note: Random key prefixes are no longer needed for performance (AWS resolved the partitioning bottleneck internally in 2018)
  • Multipart upload for large files
  • S3 Transfer Acceleration for globally distributed uploads

AWS Lambda

22. What is Lambda?

Serverless compute that runs code without managing servers. Event-driven, scales automatically, billed per invocation and duration.

23. Cold Start

Delay when AWS initializes a new execution environment before running a function. Happens when no warm container is available.

Mitigations: Provisioned Concurrency, minimize package size, reduce init code.

24. Reserved vs Provisioned Concurrency

Reserved — guarantees a maximum concurrency limit for the function.

Provisioned — keeps execution environments warm, eliminates cold starts.

25. Lambda Layers

Reusable packages containing libraries/dependencies shared across multiple Lambda functions. Max 5 layers per function.

26. Lambda limits

  • Max execution time: 15 minutes
  • Memory: 128 MB - 10 GB (CPU scales proportionally)
  • Deployment package: 50 MB (zipped), 250 MB (unzipped)

27. Asynchronous Invocation

Lambda queues the request and retries automatically on failure (up to 2 retries by default).

28. Dead Letter Queue (DLQ)

Stores failed async events in SQS or SNS after retry attempts are exhausted.

29. Lambda Destinations

Routes successful or failed async invocations to SQS, SNS, EventBridge, or another Lambda. Preferred over DLQ for async flows.

30. ECS vs Lambda — when to choose each

Lambda — short-lived, event-driven, up to 15 minutes. Zero infrastructure.

ECS — long-running processes, custom runtimes, containers needing more control over resources.

Docker & ECS

31. What is ECS?

Elastic Container Service — AWS-managed container orchestration. Runs Docker containers without managing Kubernetes.

32. ECS vs EKS

ECS — AWS-native, simpler to operate.

EKS — Managed Kubernetes, more flexible but higher operational complexity.

33. EC2 launch type vs Fargate

EC2 — you manage the underlying servers.

Fargate — serverless containers, no infrastructure management.

34. Task Definition

Blueprint specifying container image, CPU, memory, networking mode, ports, environment variables, and IAM task role.

35. ECS Task vs Service

Task — one running container instance (one-off or ephemeral).

Service — maintains the desired number of running tasks, integrates with load balancers.

36. ENTRYPOINT vs CMD

ENTRYPOINT — fixed executable that always runs.

CMD — default arguments passed to the ENTRYPOINT. Can be overridden at runtime.

37. Multi-stage Docker build

Uses multiple stages to reduce final image size by excluding build-time dependencies.

FROM python:3.11 AS builder
RUN pip install --user -r requirements.txt
 
FROM python:3.11-slim
COPY --from=builder /root/.local /root/.local
COPY . .
CMD ["python", "app.py"]

38. Why Glue instead of Lambda for ETL?

Glue is designed for large-scale Spark-based ETL. Lambda is limited to 15 min and 10 GB RAM — unsuitable for processing hundreds of millions of records.

39. Describe the ECS/ECR architecture you proposed for scalable data processing.

Experience-based question. Prepare to discuss:

  • How container images were built, tagged, and stored in ECR
  • The choice between EC2 and Fargate for the ECS launch type
  • How tasks were triggered (e.g., via Airflow or EventBridge)
  • How you handled scaling and resource allocation for data-intensive tasks

AWS Glue

40. What is AWS Glue?

Serverless ETL service for discovering, transforming, and loading data. Runs Apache Spark under the hood.

41. Glue Crawler

Automatically scans data sources and populates the Glue Data Catalog with schema metadata.

42. Glue Data Catalog

Central metadata repository — table definitions, schemas, partitions. Used by Athena, Redshift Spectrum, and EMR.

43. DynamicFrame vs DataFrame

DynamicFrame — Glue’s abstraction. Handles schema inconsistencies and semi-structured data. Convert to DataFrame for performance.

DataFrame — standard Spark abstraction with better performance and full Spark API.

44. Job Bookmark

Tracks previously processed data to enable incremental ETL — avoids reprocessing files already handled in previous runs.

45. Glue Partitioning

Organizes datasets into partitions (e.g., year/month/day) to reduce scan time and improve query performance.

46. Glue vs EMR

Glue — fully managed serverless, limited customization.

EMR — managed Hadoop/Spark clusters, full control over Spark config, better for complex or long-running jobs.

Apache Airflow / MWAA

47. What is Airflow?

Workflow orchestration platform for scheduling and managing data pipelines as Directed Acyclic Graphs (DAGs).

48. What is a DAG?

Directed Acyclic Graph — represents tasks and their execution dependencies. No cycles allowed.

49. Operator

Defines the work performed by a task (PythonOperator, BashOperator, S3ToRedshiftOperator, etc.).

50. Sensor

Waits for an external event before allowing downstream tasks to execute (e.g., S3KeySensor, ExternalTaskSensor).

51. XCom

Mechanism for passing small pieces of data between Airflow tasks. Not designed for large datasets — use S3 for that.

52. Trigger Rules

Control when downstream tasks run based on upstream outcomes. Default is all_success. Other options: all_done, one_failed, none_failed.

53. Retry Mechanism

Automatically retries failed tasks based on configured retries count and retry_delay.

54. Dynamic DAG

Generates tasks programmatically based on configuration or metadata — e.g., one task per file in an S3 prefix.

55. Airflow Pools

Limits the execution concurrency of a group of tasks to prevent overwhelming a target system (e.g., restricting API calls to 5 concurrent tasks).

56. Branch Operators

Allows the DAG to take different execution paths based on a condition (e.g., BranchPythonOperator returning the ID of the next task to execute).

57. How did your Airflow disaster recovery workflow eliminate a 48-hour manual restoration process?

Experience-based question. Prepare to discuss:

  • What the manual process entailed and why it took 48 hours
  • How you automated it using Airflow DAGs
  • What sensors, hooks, or operators were used
  • The measurable impact on reliability and recovery time

CI/CD

58. What is CI/CD?

CI — automates building and testing on every commit.

CD — automates deployment to environments after tests pass.

59. GitHub Actions

Workflow automation using YAML files (.github/workflows/). Triggers on push, PR, schedule, or manual dispatch.

60. Jenkins vs GitHub Actions

Jenkins — self-managed, highly customizable, requires maintenance.

GitHub Actions — managed, tightly integrated with GitHub, no infrastructure to run.

61. Blue-Green Deployment

Maintains two identical environments. Traffic switches to the new (Green) version after validation. Instant rollback by switching traffic back.

62. Canary Deployment

Gradually shifts a small percentage of traffic (e.g., 5%) to a new version before full rollout. Limits blast radius of failures.

63. Rollback Strategy

Restore the previous stable version when deployment validation fails. Use immutable image tags (semantic versions or commit SHAs) — never latest in production.

AWS Architecture

64. Design a scalable ETL pipeline (AWS)

Source (API/DB/S3)
    → S3 Landing Bucket
    → Airflow (orchestration)
    → Glue / ECS / Spark (transform)
    → Data Validation
    → Snowflake / Redshift
    → CloudWatch (monitoring + alerts)

Key principles: separate ingestion and transformation, store raw data first, make pipelines idempotent, partition data.

65. How would you process 700 million records?

  • Partition data (by date or entity)
  • Parallelize processing across ECS tasks or Spark
  • Use Parquet instead of CSV
  • Process incrementally, not as one monolithic job
  • Auto-scale compute
  • Retry failed partitions only

66. How do you build a highly available architecture?

  • Multi-AZ deployment
  • Auto Scaling groups
  • Application Load Balancer
  • S3 for durable object storage
  • Managed databases with Multi-AZ (RDS, Aurora)
  • Route 53 with health checks for DNS failover

67. How do you secure an AWS application?

  • IAM least privilege
  • VPC isolation + Security Groups/NACLs
  • KMS encryption at rest + TLS in transit
  • Secrets Manager for credentials
  • CloudTrail for API auditing
  • WAF + Shield for public endpoints

68. CloudWatch vs CloudTrail

CloudWatch — monitoring, metrics, logs, alarms, dashboards.

CloudTrail — records API calls and account activity for security auditing and compliance.

69. What is EventBridge?

Serverless event bus that routes events between AWS services and applications based on rules. Preferred over SNS for complex routing.

70. Cost Optimization

  • Auto Scaling to match demand
  • S3 Lifecycle policies and Intelligent-Tiering
  • Spot Instances for fault-tolerant workloads
  • Right-size EC2/ECS resources
  • Monitor with Cost Explorer and Trusted Advisor

Graph View

Backlinks