Amazon Web Services Blog

Create Snapshots From Any Block Storage Using EBS Direct APIs

I am excited to announce you can now create Amazon Elastic Block Store (EBS) snapshots from any block storage data, such as on-premises volumes, volumes from another cloud provider, existing block data stored on Amazon Simple Storage Service (S3), or even your own laptop :-) AWS customers using the cloud for disaster recovery of on-premises infrastructure all have the same question: how can I transfer my on-premises volume data to the cloud efficiently and at low cost? You usually create temporary Amazon Elastic Compute Cloud (EC2) instances, attach Amazon Elastic Block Store (EBS) volumes, transfer the data at block level from on-premises to these new Amazon Elastic Block Store (EBS) volumes, take a snapshot of every EBS volumes created and tear-down the temporary infrastructure. Some of you choose to use CloudEndure to simplify this process. Or maybe you just gave up and did not copy your on-premises volumes to the cloud because of the complexity. To simplify this, we are announcing today 3 new APIs that are part of EBS direct API, a new set of APIs we announced at re:Invent 2019. We initially launched a read and diff APIs. We extend it today with write capabilities. These 3 new APIs allow to create Amazon Elastic Block Store (EBS) snapshots from your on-premises volumes, or any block storage data that you want to be able to store and recover in AWS. With the addition of write capability in EBS direct API, you can now create new snapshots from your on-premises volumes, or create incremental snapshots, and delete them. Once a snapshot is created, it has all the benefits of snapshots created from Amazon Elastic Block Store (EBS) volumes. You can copy them, share them between AWS Accounts, keep them available for a Fast Snapshot Restore, or create Amazon Elastic Block Store (EBS) volumes from them. Having Amazon Elastic Block Store (EBS) snapshots created from any volumes, without the need to spin up Amazon Elastic Compute Cloud (EC2) instances and Amazon Elastic Block Store (EBS) volumes, allows you to simplify and to lower the cost of the creation and management of your disaster recovery copy in the cloud. Let’s have a closer look at the API You first call StartSnapshot to create a new snapshot. When the snapshot is incremental, you pass the ID of the parent snapshot. You can also pass additional tags to apply to the snapshot, or encrypt these snapshots and manage the key, just like usual. If you choose to encrypt snapshots, be sure to check our technical documentation to understand the nuances and options. Then, for each block of data, you call PutSnapshotBlock. This API has 6 mandatory parameters: snapshot-id, block-index, block-data, block-length, checksum, and checksum-algorithm. The API supports block lengths of 512 KB. You can send your blocks in any order, and in parallel, block-index keeps the order correct. After you send all the blocks, you call CompleteSnapshot with changed-blocks-count parameter having the number of blocks you sent. Let’s put all these together Here is the pseudo code you must write to create a snapshot. AmazonEBS amazonEBS = AmazonEBSClientBuilder.standard() .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(endpointName, awsRegion)) .withCredentials(credentialsProvider) .build(); response = amazonEBS.startSnapshot(startSnapshotRequest) snapshotId = response.getSnapshotId(); for each (block in changeset) { putResponse = amazonEBS.putSnapshotBlock(putSnapshotBlockRequest); } amazonEBS.completeSnapshot(completeSnapshotRequest); As usual, when using this code, you must have appropriate IAM policies allowing to call the new API. For example: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "ebs:StartSnapshot", "ebs:PutSnapshotBlock", "ebs:CompleteSnapshot" ], "Resource": "arn:aws:ec2:<Region>::snapshot/*" }] Also include some KMS related permissions when creating encrypted snapshots. In addition of the storage cost for snapshots, there is a charge per API call when you call PutSnapshotBlock. These new snapshot APIs are available in the following AWS Regions: US East (Ohio), US East (N. Virginia), US West (N. California), US West (Oregon), Asia Pacific (Hong Kong), Asia Pacific (Mumbai), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), China (Beijing), China (Ningxia), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), Europe (Stockholm), Middle East (Bahrain), and South America (São Paulo). You can start to use them today. -- seb

AWS IoT SiteWise – Now Generally Available

At AWS re:Invent 2018, we announced AWS IoT SiteWise in preview which is a fully managed AWS IoT service that you can use to collect, organize, and analyze data from industrial equipment at scale. Getting performance metrics from industrial equipment is challenging because data is often locked into proprietary on-premises data stores and typically requires specialized expertise to retrieve and place in a format that is useful for analysis. AWS IoT SiteWise simplifies this process by providing software running on a gateway that resides in your facilities and automates the process of collecting and organizing industrial equipment data. With AWS IoT SiteWise, you can easily monitor equipment across your industrial facilities to identify waste, such as breakdown of equipment and processes, production inefficiencies, and defects in products. Last year at AWS re:Invent 2019, a bunch of new features were launched including SiteWise Monitor. Today, I am excited to announce AWS IoT SiteWise is now generally available in regions of US East (N. Virginia), US West (Oregon), Europe (Frankfurt), and Europe (Ireland). Let’s see how AWS IoT SiteWise works! AWS IoT SiteWise – Getting Started You can easily explore AWS IoT SiteWise by creating a demo wind farm with a single click on the AWS IoT SiteWise console. The demo deploys an AWS CloudFormation template to create assets and generate sample data for up to a week. You can find the SiteWise demo in the upper-right corner of the AWS IoT SiteWise console, and choose Create demo. The demo takes around 3 minutes to create demo models and assets for representing a wind farm. Once you see created assets in the console, you can create virtual representations of your industrial operations with AWS IoT SiteWise assets. An asset can represent a device, a piece of equipment, or a process that uploads one or more data streams to the AWS Cloud. For example, the wind turbine that sends air temperature, propeller rotation speed, and power output time-series measurements to asset properties in AWS IoT SiteWise. Also, you can securely collect data from the plant floor from sensors, equipment, or a local on-premises gateway and upload to the AWS Cloud using a gateway software called AWS IoT SiteWise Connector. It runs on common industrial gateway devices running AWS IoT Greengrass, and reads data directly from servers and historians over the OPC Unified Architecture protocol. AWS IoT SiteWise also accepts MQTT data through AWS IoT Core, and direct ingestion using REST API. You can learn how to collect data using AWS IoT SiteWise Connector in the blog series – Part 1 of AWS IoT Blog and in the service documentation. SiteWise Monitor – Creating Managed Web Applications Once data is stored in AWS IoT SiteWise, you can stream live data in near real-time and query historical data to build downstream IoT applications, but we provide a no-code alternative with SiteWise Monitor. You can explore your library of assets, and create and share operational dashboards with plant operators for real-time monitoring and visualization of equipment health and output with SiteWise Monitor. With SiteWise Monitor console, choose Create portal to create a web application that is accessible from from a web browser on any web-enabled desktop, tablet or phone and sign-in with your corporate credentials through AWS Single Sign-On (SSO) experience. Administrators can create one or more web applications to easily share access to asset data with any team in your organization to accelerate insights. If you click a given portal link and sign in via the credential of AWS SSO, you can visualize and monitor your device, process, and equipment data to quickly identify issues and improve operational efficiency with SiteWise Monitor You can create a dashboard in a new project for your team so they can visualize and understand your project data. And, choose a visualization type that best displays your data and rearrange and resize visualizations to create a layout that fits your team’s needs. The dashboard shows asset data and computed metrics in near real time or you can compare and analyze historical time series data from multiple assets and different time periods. There is a new dashboard feature, where you can specify thresholds on the charts and have the charts change color when those thresholds are exceeded. Also, you can learn how to monitor key measurements and metrics of your assets in near-real time using SiteWise Monitor in the blog series – Part 2 of AWS IoT Blog. Furthermore, you can subscribe to the AWS IoT SiteWise modeled data via AWS IoT Core rules engine, enable condition monitoring and send notifications or alerts using AWS IoT Events in near-real time, and enable Business Intelligence (BI) reporting on historical data using Amazon QuickSight. For more detail, please refer to this hands-on guide in the blog series – Part 3 of AWS IoT Blog. Now Available! With AWS IoT SiteWise, you only pay for what you use with no minimum fees or mandatory service usage. You are billed separately for usage of messaging, data storage, data processing, and SiteWise Monitor. This approach provides you with billing transparency because you only pay for the specific AWS IoT SiteWise resources you use. Please visit the pricing page to learn more and estimate your monthly bill using the AWS IoT SiteWise Calculator. You can watch interesting talks about business cases and solutions in ‘Driving Overall Equipment Effectiveness (OEE) Across Your Industrial Facilities’ and ‘Building an End-to-End Industrial IoT (IIoT) Solution with AWS IoT‘. To learn more, please visit the AWS IoT SiteWise website or the tutorial, and developer guide. Explore AWS IoT SiteWise with Bill Vass and Cherie Wong! Please send us feedback either in the forum for AWS IoT SiteWise or through your usual AWS support contacts. – Channy;

AWS Well-Architected Framework – Updated White Papers, Tools, and Best Practices

We want to make sure that you are designing and building AWS-powered applications in the best possible way. Back in 2015 we launched AWS Well-Architected to make sure that you have all of the information that you need to do this right. The framework is built on five pillars: Operational Excellence – The ability to run and monitor systems to deliver business value and to continually improve supporting processes and procedures. Security – The ability to protect information, systems, and assets while delivering business value through risk assessments and mitigation strategies. Reliability – The ability of a system to recover from infrastructure or service disruptions, dynamically acquire computing resources to meet demand, and mitigate disruptions such as misconfigurations or transient network issues. Performance Efficiency – The ability to use computing resources efficiently to meet system requirements, and to maintain that efficiency as demand changes and technologies evolve. Cost Optimization – The ability to run systems to deliver business value at the lowest price point. Whether you are a startup, a unicorn, or an enterprise, the AWS Well-Architected Framework will point you in the right direction and then guide you along the way as you build your cloud applications. Lots of Updates Today we are making a host of updates to the Well-Architected Framework! Here’s an overview: Well-Architected Framework -This update includes new and updated questions, best practices, and improvement plans, plus additional examples and architectural considerations. We have added new best practices in operational excellence (organization), reliability (workload architecture), and cost optimization (practice Cloud Financial Management). We are also making the framework available in eight additional languages (Spanish, French, German, Japanese, Korean, Brazilian Portuguese, Simplified Chinese, and Traditional Chinese). Read the Well-Architected Framework (PDF, Kindle) to learn more. Pillar White Papers & Labs – We have updated the white papers that define each of the five pillars with additional content, including new & updated questions, real-world examples, additional cross-references, and a focus on actionable best practices. We also updated the labs that accompany each pillar: Operational Excellence (PDF, Kindle, Lab) Security (PDF, Kindle, Lab) Reliability (PDF, Kindle, Lab) Performance Efficiency (PDF, Kindle, Lab) Cost Optimization (PDF, Kindle, Lab) Well-Architected Tool – We have updated the AWS Well-Architected Tool to reflect the updates that we made to the Framework and to the White Papers. Learning More In addition to the documents that I linked above, you should also watch these videos. In this video, AWS customer Cox Automotive talks about how they are using AWS Well-Architected to deliver results across over 200 platforms: In this video, my colleague Rodney Lester tells you how to build better workloads with the Well-Architected Framework and Tool: Get Started Today If you are like me, a lot of interesting services and ideas are stashed away in a pile of things that I hope to get to “someday.” Given the importance of the five pillars that I mentioned above, I’d suggest that Well-Architected does not belong in that pile, and that you should do all that you can to learn more and to become well-architected as soon as possible! — Jeff;

New – Label Videos with Amazon SageMaker Ground Truth

Launched at AWS re:Invent 2018, Amazon Sagemaker Ground Truth is a capability of Amazon SageMaker that makes it easy to annotate machine learning datasets. Customers can efficiently and accurately label image, text and 3D point cloud data with built-in workflows, or any other type of data with custom workflows. Data samples are automatically distributed to a workforce (private, 3rd party or MTurk), and annotations are stored in Amazon Simple Storage Service (S3). Optionally, automated data labeling may also be enabled, reducing both the amount of time required to label the dataset, and the associated costs. As models become more sophisticated, AWS customers are increasingly applying machine learning prediction to video content. Autonomous driving is perhaps the most well-known use case, as safety demands that road condition and moving objects be correctly detected and tracked in real-time. Video prediction is also a popular application in Sports, tracking players or racing vehicles to compute all kinds of statistics that fans are so fond of. Healthcare organizations also use video prediction to identify and track anatomical objects in medical videos. Manufacturing companies do the same to track objects on the assembly line, parcels for logistics, and more. The list goes on, and amazing applications keep popping up in many different industries. Of course, this requires building and labeling video datasets, where objects of interest need to be labeled manually. At 30 frames per second, one minute of video translates to 1,800 individual images, so the amount of work can quickly become overwhelming. In addition, specific tools have to be built to label images, manage workflows, and so on. All this work takes valuable time and resources away from an organization’s core business. AWS customers have asked us for a better solution, and today I’m very happy to announce that Amazon Sagemaker Ground Truth now supports video labeling. Customer use case: the National Football League The National Football League (NFL) has already put this new feature to work. Says Jennifer Langton, SVP of Player Health and Innovation, NFL: “At the National Football League (NFL), we continue to look for new ways to use machine learning (ML) to help our fans, broadcasters, coaches, and teams benefit from deeper insights. Building these capabilities requires large amounts of accurately labeled training data. Amazon SageMaker Ground Truth was truly a force multiplier in accelerating our project timelines. We leveraged the new video object tracking workflow in addition to other existing computer vision (CV) labeling workflows to develop labels for training a computer vision system that tracks all 22 players as they move on the field during plays. Amazon SageMaker Ground Truth reduced the timeline for developing a high quality labeling dataset by more than 80%”. Courtesy of the NFL, here are a couple of predicted frames, showing helmet detection in a Seattle Seahawks video. This particular video has 353 frames. This first picture is frame #100. This second picture is frame #110. Introducing Video Labeling With the addition of video task types, customers can now use Amazon Sagemaker Ground Truth for: Video clip classification Video multi-frame object detection Video multi-frame object tracking The multi-frame task types support multiple labels, so that you may label different object classes present in the video frames. You can create labeling jobs to annotate frames from scratch, as well as adjustment jobs to review and fine tune frames that have already been labeled. These jobs may be distributed either to a private workforce, or to a vendor workforce you picked on AWS Marketplace. Using the built-in GUI, workers can then easily label and track objects across frames. Once they’ve annotated a frame, they can use an assistive labeling feature to predict the location of bounding boxes in the next frame, as you will see in the demo below. This significantly simplifies labeling work, saves time, and improves the quality of annotations. Last but not least, work is saved automatically. Preparing Input Data for Video Object Detection and Tracking As you would expect, input data must be located in S3. You may bring either video files, or sequences of video frames. The first option is the simplest, as Amazon Sagemaker Ground Truth includes a tool that automatically extracts frames from your video files. Optionally, you can sample frames (1 in ‘n’), in order to reduce the amount of labeling work. The extraction tool also builds a manifest file describing sequences and frames. You can learn more about it in the documentation. The second option requires two steps: extracting frames, and building the manifest file. Extracting frames can easily be performed with the popular ffmpeg open source tool. Here’s how you could convert the first 60 seconds of a video to a frame sequence. $ ffmpeg -ss 00:00:00.00 -t 00:01:0.00 -i basketball.mp4 frame%04d.jpg Each frame sequence should be uploaded to S3 under a different prefix, for example s3://my-bucket/my-videos/sequence1, s3://my-bucket/my-videos/sequence2, and so on, as explained in the documentation. Once you have uploaded your frame sequences, you may then either bring your own JSON files to describe them, or let Ground Truth crawl your sequences and build the JSON files and the manifest file for you automatically. Please note that a video sequence cannot be longer than 2,000 frames, which corresponds to about a minute of video at 30 frames per second. Each sequence should be described by a simple sequence file: A sequence number, an S3 prefix, and a number of frames. A list of frames: number, file name, and creation timestamp. Here’s an example of a sequence file. {"version": "2020-06-01", "seq-no": 1, "prefix": "s3://jsimon-smgt/videos/basketball", "number-of-frames": 1800, "frames": [ {"frame-no": 1, "frame": "frame0001.jpg", "unix-timestamp": 1594111541.71155}, {"frame-no": 2, "frame": "frame0002.jpg", "unix-timestamp": 1594111541.711552}, {"frame-no": 3, "frame": "frame0003.jpg", "unix-timestamp": 1594111541.711553}, {"frame-no": 4, "frame": "frame0004.jpg", "unix-timestamp": 1594111541.711555}, . . . Finally, the manifest file should point at the sequence files you’d like to include in the labeling job. Here’s an example. {"source-ref": "s3://jsimon-smgt/videos/seq1.json"} {"source-ref": "s3://jsimon-smgt/videos/seq2.json"} . . . Just like for other task types, the augmented manifest is available in S3 once labeling is complete. It contains annotations and labels, which you can then feed to your machine learning training job. Labeling Videos with Amazon SageMaker Ground Truth Here’s a sample video where I label the first ten frames of a sequence. You can see a screenshot below. I first use the Ground Truth GUI to carefully label the first frame, drawing bounding boxes for basketballs and basketball players. Then, I use the “Predict next” assistive labeling tool to predict the location of the boxes in the next nine frames, applying only minor adjustments to some boxes. Although this was my first try, I found the process easy and intuitive. With a little practice, I could certainly go much faster! Getting Started Now, it’s your turn. You can start labeling videos with Amazon Sagemaker Ground Truth today in the following regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Canada (Central), Europe (Ireland), Europe (London), Europe (Frankfurt), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Seoul), Asia Pacific (Sydney), Asia Pacific (Tokyo). We’re looking forward to reading your feedback. You can send it through your usual support contacts, or in the AWS Forum for Amazon SageMaker. - Julien

New – Create Amazon RDS DB Instances on AWS Outposts

Late last year I told you about AWS Outposts and invited you to Order Yours Today. As I told you at the time, this is a comprehensive, single-vendor compute and storage offering that is designed to meet the needs of customers who need local processing and very low latency in their data centers and on factory floors. Outposts uses the hardware that we use in AWS public regions I first told you about Amazon RDS back in 2009. This fully managed service makes it easy for you to launch, operate, and scale a relational database. Over the years we have added support for multiple open source and commercial databases, along with tons of features, all driven by customer requests. DB Instances on AWS Outposts Today I am happy to announce that you can now create RDS DB Instances on AWS Outposts. We are launching with support for MySQL and PostgreSQL, with plans to add other database engines in the future (as always, let us know what you need so that we can prioritize it). You can make use of important RDS features including scheduled backups to Amazon Simple Storage Service (S3), built-in encryption at rest and in transit, and more. Creating a DB Instance I can create a DB Instance using the RDS Console, API (CreateDBInstance), CLI (create-db-instance), or CloudFormation (AWS::RDS::DBInstance). I’ll use the Console, taking care to select the AWS Region that serves as “home base” for my Outpost. I open the Console and click Create database to get started: I select On-premises for the Database location, and RDS on Outposts for the On-premises database option: Next, I choose the Virtual Private Cloud (VPC). The VPC must already exist, and it must have a subnet for my Outpost. I also choose the Security Group and the Subnet: Moving forward, I select the database engine, and version. We’re launching with support for MySQL 8.0.17 and PostgreSQL 12.2-R1, with plans to add more engines and versions based on your feedback: I give my DB Instance a name (jb-database-2), and enter the credentials for the master user: Then I choose the size of the instance. I can select between Standard classes (db.m5): and Memory Optimized classes (db.r5): Next, I configure the desired amount of SSD storage: One thing to keep in mind is that each Outpost has a large, but finite amount of compute power and storage. If there’s not enough of either one free when I attempt to create the database, the request will fail. Within the Additional configuration section I can set up several database options, customize my backups, and set up the maintenance window. Once everything is ready to go, I click Create database: As usual when I use RDS, the state of my instance starts out as Creating and transitions to Available when my DB Instance is ready: After the DB instance is ready, I simply configure my code (running in my VPC or in my Outpost) to use the new endpoint: Things to Know Here are a couple of things to keep in mind about this new way to use Amazon RDS: Operations & Functions – Much of what you already know about RDS works as expected and is applicable. You can rename, reboot, stop, start, tag DB instances, and you can make use of point-in-time recovery; you can scale the instance up and down, and automatic minor version upgrades work as expected. You cannot make use of read replicas or create highly available clusters. Backup & Recover – Automated backups work as expected, and are stored in the AWS region. You can use them to create a fresh DB Instance in the cloud or in any of your Outposts. Manual snapshots also work, and are stored in the AWS region. They can be used to create a fresh DB Instance on the same Outpost. Encryption – The storage associated with your DB instance is encrypted, as are your DB snapshots, both with KMS keys. Pricing – RDS on Outposts pricing is based on a management fee that is charged on an hourly basis for each database that is managed. For more information, check out the RDS on Outposts pricing page. Available Now You can start creating RDS DB Instances on your Outposts today. — Jeff;  

Announcing the Porting Assistant for .NET

.NET Core is the future of .NET! Version 4.8 of the .NET Framework is the last major version to be released, and Microsoft has stated it will receive only bug-, reliability-, and security-related fixes going forward. For applications where you want to continue to take advantage of future investments and innovations in the .NET platform, you need to consider porting your applications to .NET Core. Also, there are additional reasons to consider porting applications to .NET Core such as benefiting from innovation in Linux and open source, improved application scaling and performance, and reducing licensing spend. Porting can, however, entail significant manual effort, some of which is undifferentiated such as updating references to project dependencies. When porting .NET Framework applications, developers need to search for compatible NuGet packages and update those package references in the application’s project files, which also need to be updated to the .NET Core project file format. Additionally, they need to discover replacement APIs since .NET Core contains a subset of the APIs available in the .NET Framework. As porting progresses, developers have to wade through long lists of compile errors and warnings to determine the best or highest priority places to continue chipping away at the task. Needless to say, this is challenging, and the added friction could be a deterrent for customers with large portfolios of applications. Today we announced the Porting Assistant for .NET, a new tool that helps customers analyze and port their .NET Framework applications to .NET Core running on Linux. The Porting Assistant for .NET assesses both the application source code and the full tree of public API and NuGet package dependencies to identify those incompatible with .NET Core and guides developers to compatible replacements when available. The suggestion engine for API and package replacements is designed to improve over time as the assistant learns more about the usage patterns and frequency of missing packages and APIs. The Porting Assistant for .NET differs from other tools in that it is able to assess the full tree of package dependencies, not just incompatible APIs. It also uses solution files as the starting point, which makes it easier to assess monolithic solutions containing large numbers of projects, instead of having to analyze and aggregate information on individual binaries. These and other abilities gives developers a jump start in the porting process. Analyzing and porting an application Getting started with porting applications using the Porting Assistant for .NET is easy, with just a couple of prerequisites. First, I need to install the .NET Core 3.1 SDK. Secondly I need a credential profile (compatible with the AWS Command Line Interface (CLI), although the CLI is not used or required). The credential profile is used to collect compatibility information on the public APIs and packages (from NuGet and core Microsoft packages) used in your application and public NuGet packages that it references. With those prerequisites taken care of, I download and run the installer for the assistant. With the assistant installed, I check out my application source code and launch the Porting Assistant for .NET from the Start menu. If I’ve previously assessed some solutions, I can view and open those from the Assessed solutions screen, enabling me to pick up where I left off. Or I can select Get started, as I’ll do here, from the home page to begin assessing my application’s solution file. I’m asked to select the credential profile I want to use, and here I can also elect to opt-in to share my telemetry data. Sharing this data helps to further improve on suggestion accuracy for all users as time goes on, and is useful in identifying issues, so we hope you consider opting-in. I click Next, browse to select the solution file that I want, and then click Assess to begin the analysis. For this post I’m going to use the open source NopCommerce project. When analysis is complete I am shown the overall results – the number of incompatible packages the application depends on, APIs it uses that are incompatible, and an overall Portability score. This score is an estimation of the effort required to port the application to .NET Core, based on the number of incompatible APIs it uses. If I’m working on porting multiple applications I can use this to identify and prioritize the applications I want to start on first. Let’s dig into the assessment overview to see what was discovered. Clicking on the solution name takes me to a more detailed dashboard and here I can see the projects that make up the application in the solution file, and for each the numbers of incompatible package and API dependencies, along with the portability score for each particular project. The current port status of each project is also listed, if I’ve already begun porting the application and have reopened the assessment. Note that with no project selected in the Projects tab the data shown in the Project references, NuGet packages, APIs, and Source files tabs is solution-wide, but I can scope the data if I wish by first selecting a project. The Project references tab shows me a graphical view of the package dependencies and I can see where the majority of the dependencies are consumed, in this case the Npp.Core, Npp.Services, and Npp.Web.Framework projects. This view can help me decide where I might want to start first, so as to get the most ‘bang for my buck’ when I begin. I can also select projects to see the specific dependencies more clearly. The NuGet packages tab gives me a look at the compatible and incompatible dependencies, and suggested replacements if available. The APIs tab lists the incompatible APIs, what package they are in, and how many times they are referenced. Source files lists all of the source files making up the projects in the application, with an indication of how many incompatible API calls can be found in each file. Selecting a source file opens a view showing me where the incompatible APIs are being used and suggested package versions to upgrade to, if they exist, to resolve the issue. If there is no suggested replacement by simply updating to a different package version then I need to crack open a source editor and update the code to use a different API or approach. Here I’m looking at the report for DependencyRegistrar.cs, that exists in the Nop.Web project, and uses the Autofac NuGet package. Let’s start porting the application, starting with the Nop.Core project. First, I navigate back to the Projects tab, select the project, and then click Port project. During porting the tool will help me update project references to NuGet packages, and also updates the project files themselves to the newer .NET Core formats. I have the option of either making a copy of the application’s solution file, project files, and source files, or I can have the changes made in-place. Here I’ve elected to make a copy. Clicking Save copies the application source code to the selected location, and opens the Port projects view where I can set the new target framework version (in this case netcoreapp3.1), and the list of NuGet dependencies for the project that I need to upgrade. For each incompatible package the Porting Assistant for .NET gives me a list of possible version upgrades and for each version, I am shown the number of incompatible APIs that will either remain, or will additionally become incompatible. For the package I selected here there’s no difference but for cases where later versions potentially increase the number of incompatible APIs that I would then need to manually fix up in the source code, this indication helps me to make a trade-off decision around whether to upgrade to the latest versions of a package, or stay with an older one. Once I select a version, the Deprecated API calls field alongside the package will give me a reminder of what I need to fix up in a code editor. Clicking on the value summarizes the deprecated calls. I continue with this process for each package dependency and when I’m ready, click Port to have the references updated. Using my IDE, I can then go into the source files and work on replacing the incompatible API calls using the Porting Assistant for .NET‘s source file and deprecated API list views as a reference, and follow a similar process for the other projects in my application. Improving the suggestion engine The suggestion engine behind the Porting Assistant for .NET is designed to learn and give improved results over time, as customers opt-in to sharing their telemetry. The data models behind the engine, which are the result of analyzing hundreds of thousands of unique packages with millions of package versions, are available on GitHub. We hope that you’ll consider helping improve accuracy and completeness of the results by contributing your data. The user guide gives more details on how the data is used. The Porting Assistant for .NET is free to use and is available now. — Steve

AWS App2Container – A New Containerizing Tool for Java and ASP.NET Applications

Our customers are increasingly developing their new applications with containers and serverless technologies, and are using modern continuous integration and delivery (CI/CD) tools to automate the software delivery life cycle. They also maintain a large number of existing applications that are built and managed manually or using legacy systems. Maintaining these two sets of applications with disparate tooling adds to operational overhead and slows down the pace of delivering new business capabilities. As much as possible, they want to be able to standardize their management tooling and CI/CD processes across both their existing and new applications, and see the option of packaging their existing applications into containers as the first step towards accomplishing that goal. However, containerizing existing applications requires a long list of manual tasks such as identifying application dependencies, writing dockerfiles, and setting up build and deployment processes for each application. These manual tasks are time consuming, error prone, and can slow down the modernization efforts. Today, we are launching AWS App2Container, a new command-line tool that helps containerize existing applications that are running on-premises, in Amazon Elastic Compute Cloud (EC2), or in other clouds, without needing any code changes. App2Container discovers applications running on a server, identifies their dependencies, and generates relevant artifacts for seamless deployment to Amazon ECS and Amazon EKS. It also provides integration with AWS CodeBuild and AWS CodeDeploy to enable a repeatable way to build and deploy containerized applications. AWS App2Container generates the following artifacts for each application component: Application artifacts such as application files/folders, Dockerfiles, container images in Amazon Elastic Container Registry (ECR), ECS Task definitions, Kubernetes deployment YAML, CloudFormation templates to deploy the application to Amazon ECS or EKS, and templates to set up a build/release pipeline in AWS Codepipeline which also leverages AWS CodeBuild and CodeDeploy. Starting today, you can use App2Container to containerize ASP.NET (.NET 3.5+) web applications running in IIS 7.5+ on Windows, and Java applications running on Linux—standalone JBoss, Apache Tomcat, and generic Java applications such as Spring Boot, IBM WebSphere, Oracle WebLogic, etc. By modernizing existing applications using containers, you can make them portable, increase development agility, standardize your CI/CD processes, and reduce operational costs. Now let’s see how it works! AWS App2Container – Getting Started AWS App2Container requires that the following prerequisites be installed on the server(s) hosting your application: AWS Command Line Interface (CLI) version 1.14 or later, Docker tools, and (in the case of ASP.NET) Powershell 5.0+ for applications running on Windows. Additionally, you need to provide appropriate IAM permissions to App2Container to interact with AWS services. For example, let’s look how you containerize your existing Java applications. App2Container CLI for Linux is packaged as a tar.gz archive. The file provides users an interactive shell script, install.sh to install the App2Container CLI. Running the script guides users through the install steps and also updates the user’s path to include the App2Container CLI commands. First, you can begin by running a one-time initialization on the installed server for the App2Container CLI with the init command. $ sudo app2container init Workspace directory path for artifacts[default: /home/ubuntu/app2container/ws]: AWS Profile (configured using 'aws configure --profile')[default: default]: Optional S3 bucket for application artifacts (Optional)[default: none]: Report usage metrics to AWS? (Y/N)[default: y]: Require images to be signed using Docker Content Trust (DCT)? (Y/N)[default: n]: Configuration saved This sets up a workspace to store application containerization artifacts (minimum 20GB of disk space available). You can extract them into your Amazon Simple Storage Service (S3) bucket using your AWS profile configured to use AWS services. Next, you can view Java processes that are running on the application server by using the inventory command. Each Java application process has a unique identifier (for example, java-tomcat-9e8e4799) which is the application ID. You can use this ID to refer to the application with other App2Container CLI commands. $ sudo app2container inventory { "java-jboss-5bbe0bec": { "processId": 27366, "cmdline": "java ... /home/ubuntu/wildfly-10.1.0.Final/modules org.jboss.as.standalone -Djboss.home.dir=/home/ubuntu/wildfly-10.1.0.Final -Djboss.server.base.dir=/home/ubuntu/wildfly-10.1.0.Final/standalone ", "applicationType": "java-jboss" }, "java-tomcat-9e8e4799": { "processId": 2537, "cmdline": "/usr/bin/java ... -Dcatalina.home=/home/ubuntu/tomee/apache-tomee-plume-7.1.1 -Djava.io.tmpdir=/home/ubuntu/tomee/apache-tomee-plume-7.1.1/temp org.apache.catalina.startup.Bootstrap start ", "applicationType": "java-tomcat" } } You can also intialize ASP.NET applications on an administrator-run PowerShell session of Windows Servers with IIS version 7.0 or later. Note that Docker tools and container support are available on Windows Server 2016 and later versions. You can select to run all app2container operations on the application server with Docker tools installed or use a worker machine with Docker tools using Amazon ECS-optimized Windows Server AMIs. PS> app2container inventory { "iis-smarts-51d2dbf8": { "siteName": "nopCommerce39", "bindings": "http/*:90:", "applicationType": "iis" } } The inventory command displays all IIS websites on the application server that can be containerized. Each IIS website process has a unique identifier (for example, iis-smarts-51d2dbf8) which is the application ID. You can use this ID to refer to the application with other App2Container CLI commands. You can choose a specific application by referring to its application ID and generate an analysis report for the application by using the analyze command. $ sudo app2container analyze --application-id java-tomcat-9e8e4799 Created artifacts folder /home/ubuntu/app2container/ws/java-tomcat-9e8e4799 Generated analysis data in /home/ubuntu/app2container/ws/java-tomcat-9e8e4799/analysis.json Analysis successful for application java-tomcat-9e8e4799 Please examine the same, make appropriate edits and initiate containerization using "app2container containerize --application-id java-tomcat-9e8e4799" You can use the analysis.json template generated by the application analysis to gather information on the analyzed application that helps identify all system dependencies from the analysisInfo section, and update containerization parameters to customize the container images generated for the application using the containerParameters section. $ cat java-tomcat-9e8e4799/analysis.json { "a2CTemplateVersion": "1.0", "createdTime": "2020-06-24 07:40:5424", "containerParameters": { "_comment1": "*** EDITABLE: The below section can be edited according to the application requirements. Please see the analyisInfo section below for deetails discoverd regarding the application. ***", "imageRepository": "java-tomcat-9e8e4799", "imageTag": "latest", "containerBaseImage": "ubuntu:18.04", "coopProcesses": [ 6446, 6549, 6646] }, "analysisInfo": { "_comment2": "*** NON-EDITABLE: Analysis Results ***", "processId": 2537 "appId": "java-tomcat-9e8e4799", "userId": "1000", "groupId": "1000", "cmdline": [...], "os": {...}, "ports": [...] } } Also, you can run the $ app2container extract --application-id java-tomcat-9e8e4799 command to generate an application archive for the analyzed application. This depends on the analysis.json file generated earlier in the workspace folder for the application,and adheres to any containerization parameter updates specified in there. By using extract command, you can continue the workflow on a worker machine after running the first set of commands on the application server. Now you can containerize command generated Docker images for the selected application. $ sudo app2container containerize --application-id java-tomcat-9e8e4799 AWS pre-requisite check succeeded Docker pre-requisite check succeeded Extracted container artifacts for application Entry file generated Dockerfile generated under /home/ubuntu/app2container/ws/java-tomcat-9e8e4799/Artifacts Generated dockerfile.update under /home/ubuntu/app2container/ws/java-tomcat-9e8e4799/Artifacts Generated deployment file at /home/ubuntu/app2container/ws/java-tomcat-9e8e4799/deployment.json Containerization successful. Generated docker image java-tomcat-9e8e4799 You're all set to test and deploy your container image. Next Steps: 1. View the container image with \"docker images\" and test the application. 2. When you're ready to deploy to AWS, please edit the deployment file as needed at /home/ubuntu/app2container/ws/java-tomcat-9e8e4799/deployment.json. 3. Generate deployment artifacts using app2container generate app-deployment --application-id java-tomcat-9e8e4799 Using this command, you can view the generated container images using Docker images on the machine where the containerize command is run. You can use the docker run command to launch the container and test application functionality. Note that in addition to generating container images, the containerize command also generates a deployment.json template file that you can use with the next generate-appdeployment command. You can edit the parameters in the deployment.json template file to change the image repository name to be registered in Amazon ECR, the ECS task definition parameters, or the Kubernetes App name. $ cat java-tomcat-9e8e4799/deployment.json { "a2CTemplateVersion": "1.0", "applicationId": "java-tomcat-9e8e4799", "imageName": "java-tomcat-9e8e4799", "exposedPorts": [ { "localPort": 8090, "protocol": "tcp6" } ], "environment": [], "ecrParameters": { "ecrRepoTag": "latest" }, "ecsParameters": { "createEcsArtifacts": true, "ecsFamily": "java-tomcat-9e8e4799", "cpu": 2, "memory": 4096, "dockerSecurityOption": "", "enableCloudwatchLogging": false, "publicApp": true, "stackName": "a2c-java-tomcat-9e8e4799-ECS", "reuseResources": { "vpcId": "", "cfnStackName": "", "sshKeyPairName": "" }, "gMSAParameters": { "domainSecretsArn": "", "domainDNSName": "", "domainNetBIOSName": "", "createGMSA": false, "gMSAName": "" } }, "eksParameters": { "createEksArtifacts": false, "applicationName": "", "stackName": "a2c-java-tomcat-9e8e4799-EKS", "reuseResources": { "vpcId": "", "cfnStackName": "", "sshKeyPairName": "" } } } At this point, the application workspace where the artifacts are generated serves as an iteration sandbox. You can choose to edit the Dockerfile generated here to make changes to their application and use the docker build command to build new container images as needed. You can generate the artifacts needed to deploy the application containers in Amazon EKS by using the generate-deployment command. $ sudo app2container generate app-deployment --application-id java-tomcat-9e8e4799 AWS pre-requisite check succeeded Docker pre-requisite check succeeded Created ECR Repository Uploaded Cloud Formation resources to S3 Bucket: none Generated Cloud Formation Master template at: /home/ubuntu/app2container/ws/java-tomcat-9e8e4799/EksDeployment/amazon-eks-master.template.yaml EKS Cloudformation templates and additional deployment artifacts generated successfully for application java-tomcat-9e8e4799 You're all set to use AWS Cloudformation to manage your application stack. Next Steps: 1. Edit the cloudformation template as necessary. 2. Create an application stack using the AWS CLI or the AWS Console. AWS CLI command: aws cloudformation deploy --template-file /home/ubuntu/app2container/ws/java-tomcat-9e8e4799/EksDeployment/amazon-eks-master.template.yaml --capabilities CAPABILITY_NAMED_IAM --stack-name java-tomcat-9e8e4799 3. Setup a pipeline for your application stack: app2container generate pipeline --application-id java-tomcat-9e8e4799 This command works based on the deployment.json template file produced as part of running the containerize command. App2Container will now generate ECS/EKS cloudformation templates as well and an option to deploy those stacks. The command registers the container image to user specified ECR repository, generates cloudformation template for Amazon ECS and EKS deployments. You can register ECS task definition with Amazon ECS and use kubectl to launch the containerized application on the existing Amazon EKS or self-managed kubernetes cluster using App2Container generated amazon-eks-master.template.deployment.yaml. Alternatively, you can directly deploy containerized applications by --deploy options into Amazon EKS. $ sudo app2container generate app-deployment --application-id java-tomcat-9e8e4799 --deploy AWS pre-requisite check succeeded Docker pre-requisite check succeeded Created ECR Repository Uploaded Cloud Formation resources to S3 Bucket: none Generated Cloud Formation Master template at: /home/ubuntu/app2container/ws/java-tomcat-9e8e4799/EksDeployment/amazon-eks-master.template.yaml Initiated Cloudformation stack creation. This may take a few minutes. Please visit the AWS Cloudformation Console to track progress. Deploying application to EKS Handling ASP.NET Applications with Windows Authentication Containerizing ASP.NET applications is almost same process as Java applications, but Windows containers cannot be directly domain joined. They can however still use Active Directory (AD) domain identities to support various authentication scenarios. App2Container detects if a site is using Windows authentication and accordingly makes the IIS site’s application pool run as the network service identity, and generates the new cloudformation templates for Windows authenticated IIS applications. The creation of gMSA and AD Security group, domain join ECS nodes and making containers use this gMSA are all taken care of by those templates. Also, it provides two PowerShell scripts as output to the $ app2container containerize command along with an instruction file on how to use it. The following is an example output: PS C:\Windows\system32> app2container containerize --application-id iis-SmartStoreNET-a726ba0b Running AWS pre-requisite check... Running Docker pre-requisite check... Container build complete. Please use "docker images" to view the generated container images. Detected that the Site is using Windows Authentication. Generating powershell scripts into C:\Users\Admin\AppData\Local\app2container\iis-SmartStoreNET-a726ba0b\Artifacts required to setup Container host with Windows Authentication Please look at C:\Users\Admin\AppData\Local\app2container\iis-SmartStoreNET-a726ba0b\Artifacts\WindowsAuthSetupInstructions.md for setup instructions on Windows Authentication. A deployment file has been generated under C:\Users\Admin\AppData\Local\app2container\iis-SmartStoreNET-a726ba0b Please edit the same as needed and generate deployment artifacts using "app2container generate-deployment" The first PowerShellscript, DomainJoinAddToSecGroup.ps1, joins the container host and adds it to an Active Directory security group. The second script, CreateCredSpecFile.ps1, creates a Group Managed Service Account (gMSA), grants access to the Active Directory security group, generates the credential spec for this gMSA, and stores it locally on the container host. You can execute these PowerShellscripts on the ECS host. The following is an example usage of the scripts: PS C:\Windows\system32> .\DomainJoinAddToSecGroup.ps1 -ADDomainName Dominion.com -ADDNSIp 10.0.0.1 -ADSecurityGroup myIISContainerHosts -CreateADSecurityGroup:$true PS C:\Windows\system32> .\CreateCredSpecFile.ps1 -GMSAName MyGMSAForIIS -CreateGMSA:$true -ADSecurityGroup myIISContainerHosts Before executing the app2container generate-deployment command, edit the deployment.json file to change the value of dockerSecurityOption to the name of the CredentialSpec file that the CreateCredSpecFile script generated. For example, "dockerSecurityOption": "credentialspec:file://dominion_mygmsaforiis.json" Effectively, any access to network resource made by the IIS server inside the container for the site will now use the above gMSA to authenticate. The final step is to authorize this gMSA account on the network resources that the IIS server will access. A common example is authorizing this gMSA inside the SQL Server. Finally, if the application must connect to a database to be fully functional and you run the container in Amazon ECS, ensure that the application container created from the Docker image generated by the tool has connectivity to the same database. You can refer to this documentation for options on migrating: MS SQL Server from Windows to Linux on AWS, Database Migration Service, and backup and restore your MS SQL Server to Amazon RDS. Now Available AWS App2Container is offered free. You only pay for the actual usage of AWS services like Amazon EC2, ECS, EKS, and S3 etc based on their usage. For details, please refer to App2Container FAQs and documentations. Give this a try, and please send us feedback either through your usual AWS Support contacts, on the AWS Forum for ECS, AWS Forum for EKS, or on the container roadmap on Github. — Channy;

Amazon RDS Proxy – Now Generally Available

At AWS re:Invent 2019, we launched the preview of Amazon RDS Proxy, a fully managed, highly available database proxy for Amazon Relational Database Service (RDS) that makes applications more scalable, more resilient to database failures, and more secure. Following the preview of MySQL engine, we extended to the PostgreSQL compatibility. Today, I am pleased to announce that we are now generally available for both engines. Many applications, including those built on modern serverless architectures using AWS Lambda, Fargate, Amazon ECS, or EKS can have a large number of open connections to the database server, and may open and close database connections at a high rate, exhausting database memory and compute resources. Amazon RDS Proxy allows applications to pool and share connections established with the database, improving database efficiency, application scalability, and security. With RDS Proxy, failover times for Amazon Aurora and RDS databases are reduced by up to 66%, and database credentials, authentication, and access can be managed through integration with AWS Secrets Manager and AWS Identity and Access Management (IAM). Amazon RDS Proxy can be enabled for most applications with no code change, and you don’t need to provision or manage any additional infrastructure and only pay per vCPU of the database instance for which the proxy is enabled. Amazon RDS Proxy – Getting started You can get started with Amazon RDS Proxy in just a few clicks by going to the AWS management console and creating an RDS Proxy endpoint for your RDS databases. In the navigation pane, choose Proxies and Create proxy. You can also see the proxy panel below. To create your proxy, specify the Proxy identifier, a unique name of your choosing, and choose the database engine – either MySQL or PostgreSQL. Choose the encryption setting if you want the proxy to enforce TLS / SSL for all connection between application and proxy, and specify a time period that a client connection can be idle before the proxy can close it. A client connection is considered idle when the application doesn’t submit a new request within the specified time after the previous request completed. The underlying connection between the proxy and database stays open and is returned to the connection pool. Thus, it’s available to be reused for new client connections. Next, choose one RDS DB instance or Aurora DB cluster in Database to access through this proxy. The list only includes DB instances and clusters with compatible database engines, engine versions, and other settings. Specify Connection pool maximum connections, a value between 1 and 100. This setting represents the percentage of the max_connections value that RDS Proxy can use for its connections. If you only intend to use one proxy with this DB instance or cluster, you can set it to 100. For details about how RDS Proxy uses this setting, see Connection Limits and Timeouts. Choose at least one Secrets Manager secret associated with the RDS DB instance or Aurora DB cluster that you intend to access with this proxy, and select an IAM role that has permission to access the Secrets Manager secrets you chose. If you don’t have an existing secret, please click Create a new secret before setting up the RDS proxy. After setting VPC Subnets and a security group, please click Create proxy. If you more settings in details, please refer to the documentation. You can see the new RDS proxy after waiting a few minutes and then point your application to the RDS Proxy endpoint. That’s it! You can also create an RDS proxy easily via AWS CLI command. aws rds create-db-proxy \ --db-proxy-name channy-proxy \ --role-arn iam_role \ --engine-family { MYSQL|POSTGRESQL } \ --vpc-subnet-ids space_separated_list \ [--vpc-security-group-ids space_separated_list] \ [--auth ProxyAuthenticationConfig_JSON_string] \ [--require-tls | --no-require-tls] \ [--idle-client-timeout value] \ [--debug-logging | --no-debug-logging] \ [--tags comma_separated_list] How RDS Proxy works Let’s see an example that demonstrates how open connections continue working during a failover when you reboot a database or it becomes unavailable due to a problem. This example uses a proxy named channy-proxy and an Aurora DB cluster with DB instances instance-8898 and instance-9814. When the failover-db-cluster command is run from the Linux command line, the writer instance that the proxy is connected to changes to a different DB instance. You can see that the DB instance associated with the proxy changes while the connection remains open. $ mysql -h channy-proxy.proxy-abcdef123.us-east-1.rds.amazonaws.com -u admin_user -p Enter password: ... mysql> select @@aurora_server_id; +--------------------+ | @@aurora_server_id | +--------------------+ | instance-9814 | +--------------------+ 1 row in set (0.01 sec) mysql> [1]+ Stopped mysql -h channy-proxy.proxy-abcdef123.us-east-1.rds.amazonaws.com -u admin_user -p $ # Initially, instance-9814 is the writer. $ aws rds failover-db-cluster --db-cluster-id cluster-56-2019-11-14-1399 JSON output $ # After a short time, the console shows that the failover operation is complete. $ # Now instance-8898 is the writer. $ fg mysql -h channy-proxy.proxy-abcdef123.us-east-1.rds.amazonaws.com -u admin_user -p mysql> select @@aurora_server_id; +--------------------+ | @@aurora_server_id | +--------------------+ | instance-8898 | +--------------------+ 1 row in set (0.01 sec) mysql> [1]+ Stopped mysql -h channy-proxy.proxy-abcdef123.us-east-1.rds.amazonaws.com -u admin_user -p $ aws rds failover-db-cluster --db-cluster-id cluster-56-2019-11-14-1399 JSON output $ # After a short time, the console shows that the failover operation is complete. $ # Now instance-9814 is the writer again. $ fg mysql -h channy-proxy.proxy-abcdef123.us-east-1.rds.amazonaws.com -u admin_user -p mysql> select @@aurora_server_id; +--------------------+ | @@aurora_server_id | +--------------------+ | instance-9814 | +--------------------+ 1 row in set (0.01 sec) +---------------+---------------+ | Variable_name | Value | +---------------+---------------+ | hostname | ip-10-1-3-178 | +---------------+---------------+ 1 row in set (0.02 sec) With RDS Proxy, you can build applications that can transparently tolerate database failures without needing to write complex failure handling code. RDS Proxy automatically routes traffic to a new database instance while preserving application connections. You can review the demo for an overview of RDS Proxy and the steps you need take to access RDS Proxy from a Lambda function. If you want to know how your serverless applications maintain excellent performance even at peak loads, please read this blog post. For a deeper dive into using RDS Proxy for MySQL with serverless, visit this post. The following are a few things that you should be aware of: Currently, RDS Proxy is available for the MySQL and PostgreSQL engine family. This engine family includes RDS for MySQL 5.6 and 5.7, PostgreSQL 10.11 and 11.5. In an Aurora cluster, all of the connections in the connection pool are handled by the Aurora primary instance. To perform load balancing for read-intensive workloads, you still use the reader endpoint directly for the Aurora cluster. Your RDS Proxy must be in the same VPC as the database. Although the database can be publicly accessible, the proxy can’t be. Proxies don’t support compressed mode. For example, they don’t support the compression used by the --compress or -C options of the mysql command. Now Available! Amazon RDS Proxy is generally available in US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon), Europe (Frankfurt), Europe (Ireland), Europe (London) , Asia Pacific (Mumbai), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney) and Asia Pacific (Tokyo) regions for Aurora MySQL, RDS for MySQL, Aurora PostgreSQL, and RDS for PostgreSQL, and it includes support for Aurora Serverless and Aurora Multi-Master. Take a look at the product page, pricing, and the documentation to learn more. Please send us feedback either in the AWS forum for Amazon RDS or through your usual AWS support contacts. – Channy;

Find Your Most Expensive Lines of Code – Amazon CodeGuru Is Now Generally Available

Bringing new applications into production, maintaining their code base as they grow and evolve, and at the same time respond to operational issues, is a challenging task. For this reason, you can find many ideas on how to structure your teams, on which methodologies to apply, and how to safely automate your software delivery pipeline. At re:Invent last year, we introduced in preview Amazon CodeGuru, a developer tool powered by machine learning that helps you improve your applications and troubleshoot issues with automated code reviews and performance recommendations based on runtime data. During the last few months, many improvements have been launched, including a more cost-effective pricing model, support for Bitbucket repositories, and the ability to start the profiling agent using a command line switch, so that you no longer need to modify the code of your application, or add dependencies, to run the agent. You can use CodeGuru in two ways: CodeGuru Reviewer uses program analysis and machine learning to detect potential defects that are difficult for developers to find, and recommends fixes in your Java code. The code can be stored in GitHub (now also in GitHub Enterprise), AWS CodeCommit, or Bitbucket repositories. When you submit a pull request on a repository that is associated with CodeGuru Reviewer, it provides recommendations for how to improve your code. Each pull request corresponds to a code review, and each code review can include multiple recommendations that appear as comments on the pull request. CodeGuru Profiler provides interactive visualizations and recommendations that help you fine-tune your application performance and troubleshoot operational issues using runtime data from your live applications. It currently supports applications written in Java virtual machine (JVM) languages such as Java, Scala, Kotlin, Groovy, Jython, JRuby, and Clojure. CodeGuru Profiler can help you find the most expensive lines of code, in terms of CPU usage or introduced latency, and suggest ways you can improve efficiency and remove bottlenecks. You can use CodeGuru Profiler in production, and when you test your application with a meaningful workload, for example in a pre-production environment. Today, Amazon CodeGuru is generally available with the addition of many new features. In CodeGuru Reviewer, we included the following: Support for Github Enterprise – You can now scan your pull requests and get recommendations against your source code on Github Enterprise on-premises repositories, together with a description of what’s causing the issue and how to remediate it. New types of recommendations to solve defects and improve your code – For example, checking input validation, to avoid issues that can compromise security and performance, and looking for multiple copies of code that do the same thing. In CodeGuru Profiler, you can find these new capabilities: Anomaly detection – We automatically detect anomalies in the application profile for those methods that represent the highest proportion of CPU time or latency. Lambda function support – You can now profile AWS Lambda functions just like applications hosted on Amazon Elastic Compute Cloud (EC2) and containerized applications running on Amazon ECS and Amazon Elastic Kubernetes Service, including those using AWS Fargate. Cost of issues in the recommendation report – Recommendations contain actionable resolution steps which explain what the problem is, the CPU impact, and how to fix the issue. To help you better prioritize your activities, you now have an estimation of the savings introduced by applying the recommendation. Color-my-code – In the visualizations, to help you easily find your own code, we are coloring your methods differently from frameworks and other libraries you may use. CloudWatch metrics and alerts – To keep track and monitor efficiency issues that have been discovered. Let’s see some of these new features at work! Using CodeGuru Reviewer with a Lambda Function I create a new repo in my GitHub account, and leave it empty for now. Locally, where I am developing a Lambda function using the Java 11 runtime, I initialize my Git repo and add only the README.md file to the master branch. In this way, I can add all the code as a pull request later and have it go through a code review by CodeGuru. git init git add README.md git commit -m "First commit" Now, I add the GitHub repo as origin, and push my changes to the new repo: git remote add origin https://github.com/<my-user-id>/amazon-codeguru-sample-lambda-function.git git push -u origin master I associate the repository in the CodeGuru console: When the repository is associated, I create a new dev branch, add all my local files to it, and push it remotely: git checkout -b dev git add . git commit -m "Code added to the dev branch" git push --set-upstream origin dev In the GitHub console, I open a new pull request by comparing changes across the two branches, master and dev. I verify that the pull request is able to merge, then I create it. Since the repository is associated with CodeGuru, a code review is listed as Pending in the Code reviews section of the CodeGuru console. After a few minutes, the code review status is Completed, and CodeGuru Reviewer issues a recommendation on the same GitHub page where the pull request was created. Oops! I am creating the Amazon DynamoDB service object inside the function invocation method. In this way, it cannot be reused across invocations. This is not efficient. To improve the performance of my Lambda function, I follow the CodeGuru recommendation, and move the declaration of the DynamoDB service object to a static final attribute of the Java application object, so that it is instantiated only once, during function initialization. Then, I follow the link in the recommendation to learn more best practices for working with Lambda functions. Using CodeGuru Profiler with a Lambda Function In the CodeGuru console, I create a MyServerlessApp-Development profiling group and select the Lambda compute platform. Next, I give the AWS Identity and Access Management (IAM) role used by my Lambda function permissions to submit data to this profiling group. Now, the console is giving me all the info I need to profile my Lambda function. To configure the profiling agent, I use a couple of environment variables: AWS_CODEGURU_PROFILER_GROUP_ARN to specify the ARN of the profiling group to use. AWS_CODEGURU_PROFILER_ENABLED to enable (TRUE) or disable (FALSE) profiling. I follow the instructions (for Maven and Gradle) to add a dependency, and include the profiling agent in the build. Then, I update the code of the Lambda function to wrap the handler function inside the LambdaProfiler provided by the agent. To generate some load, I start a few scripts invoking my function using the Amazon API Gateway as trigger. After a few minutes, the profiling group starts to show visualizations describing the runtime behavior of my Lambda function. For example, I can see how much CPU time is spent in the different methods of my function. At the bottom, there are the entry point methods. As I scroll up, I find methods that are called deeper in the stack trace. I right-click and hide the LambdaRuntimeClient methods to focus on my code. Note that my methods are colored differently than those in the packages I am using, such as the AWS SDK for Java. I am mostly interested in what happens in the handler method invoked by the Lambda platform. I select the handler method, and now it becomes the new “base” of the visualization. As I move my pointer on each of my methods, I get more information, including an estimation of the yearly cost of running that specific part of the code in production, based on the load experienced by the profiling agent during the selected time window. In my case, the handler function cost is estimated to be $6. If I select the two main functions above, I have an estimation of $3 each. The cost estimation works for code running on Lambda functions, EC2 instances, and containerized applications. Similarly, I can visualize Latency, to understand how much time is spent inside the methods in my code. I keep the Lambda function handler method selected to drill down into what is under my control, and see where time is being spent the most. The CodeGuru Profiler is also providing a recommendation based on the data collected. I am spending too much time (more than 4%) in managing encryption. I can use a more efficient crypto provider, such as the open source Amazon Corretto Crypto Provider, described in this blog post. This should lower the time spent to what is expected, about 1% of my profile. Finally, I edit the profiling group to enable notifications. In this way, if CodeGuru detects an anomaly in the profile of my application, I am notified in one or more Amazon Simple Notification Service (SNS) topics. Available Now Amazon CodeGuru is available today in 10 regions, and we are working to add more regions in the coming months. For regional availability, please see the AWS Region Table. CodeGuru helps you improve your application code and reduce compute and infrastructure costs with an automated code reviewer and application profiler that provide intelligent recommendations. Using visualizations based on runtime data, you can quickly find the most expensive lines of code of your applications. With CodeGuru, you pay only for what you use. Pricing is based on the lines of code analyzed by CodeGuru Reviewer, and on sampling hours for CodeGuru Profiler. To learn more, please see the documentation or check out the video by Jeff! — Danilo

Introducing Amazon Honeycode – Build Web & Mobile Apps Without Writing Code

VisiCalc was launched in 1979, and I purchased a copy (shown at right) for my Apple II. The spreadsheet model was clean, easy to use, and most of all, easy to teach. I was working in a retail computer store at that time, and knew that this product was a big deal when customers started asking to purchase the software, and for whatever hardware that was needed to run it. Today’s spreadsheets fill an important gap between mass-produced packaged applications and custom-built code created by teams of dedicated developers. Every tool has its limits, however. Sharing data across multiple users and multiple spreadsheets is difficult, as is dealing with large amounts of data. Integration & automation are also challenging, and require specialized skills. In many cases, those custom-built apps would be a better solution than a spreadsheet, but a lack of developers or other IT resources means that these apps rarely get built. Introducing Amazon Honeycode Today we are launching Amazon Honeycode in beta form. This new fully-managed AWS service gives you the power to build powerful mobile & web applications without writing any code. It uses the familiar spreadsheet model and lets you get started in minutes. If you or your teammates are already familiar with spreadsheets and formulas, you’ll be happy to hear that just about everything you know about sheets, tables, values, and formulas still applies. Amazon Honeycode includes templates for some common applications that you and other members of your team can use right away: You can customize these apps at any time and the changes will be deployed immediately. You can also start with an empty table, or by importing some existing data in CSV form. The applications that you build with Honeycode can make use of a rich palette of user interface objects including lists, buttons, and input fields: You can also take advantage of a repertoire of built-in, trigger-driven actions that can generate email notifications and modify tables: Honeycode also includes a lengthy list of built-in functions. The list includes many functions that will be familiar to users of existing spreadsheets, along with others that are new to Honeycode. For example, FindRow is a more powerful version of the popular Vlookup function. Getting Started with Honeycode It is easy to get started. I visit the Honeycode Builder, and create my account: After logging in I see My Drive, with my workbooks & apps, along with multiple search, filter, & view options: I can open & explore my existing items, or I can click Create workbook to make something new. I do that, and then select the Simple To-do template: The workbook, tables, and the apps are created and ready to use right away. I can simply clear the sample data from the tables and share the app with the users, or I can inspect and customize it. Let’s inspect it first, and then share it! After I create the new workbook, the Tasks table is displayed and I can see the sample data: Although this looks like a traditional spreadsheet, there’s a lot going on beneath the surface. Let’s go through, column-by-column: A (Task) – Plain text. B (Assignee) – Text, formatted as a Contact. C (First Name) – Text, computed by a formula: In the formula, Assignee refers to column B, and First Name refers to the first name of the contact. D (Due) – A date, with multiple formatting options: E (Done) – A picklist that pulls values from another table, and that is formatted as a Honeycode rowlink. Together, this restricts the values in this column to those found in the other table (Done, in this case, with the values Yes and No), and also makes the values from that table visible within the context of this one: F (Remind On) – Another picklist, this one taking values from the ReminderOptions table: G (Notification) – Another date. This particular table uses just a few of the features and options that are available to you. I can use the icons on the left to explore my workbook: I can see the tables: I can also see the apps. A single Honeycode workbook can contain multiple apps that make use of the same tables: I’ll return to the apps and the App Builder in a minute, but first I’ll take a look at the automations: Again, all of the tables and apps in the workbook can use any of the automations in the workbook. The Honeycode App Builder Let’s take a closer look at the app builder. As was the case with the tables, I will show you some highlights and let you explore the rest yourself. Here’s what I see when I open my Simple To-do app in the builder: This app contains four screens (My Tasks, All Tasks, Edit, and Add Task). All screens have both web and mobile layouts. Newly created screens, and also those in this app, have the layouts linked, so that changes to one are reflected in the other. I can unlink the layouts if I want to exercise more control over the controls, the presentation, or to otherwise differentiate the two: Objects within a screen can reference data in tables. For example, the List object on the My Task screen filters rows of the Tasks table, selecting the undone tasks and ordering them by the due date: Here’s the source expression: =Filter(Tasks,"Tasks[Done]<>% ORDER BY Tasks[Due]","Yes") The “%”  in the filter condition is replaced by the second parameter (“Yes”) when the filter is evaluated. This substitution system makes it easy for you to create interesting & powerful filters using the FILTER() function. When the app runs, the objects within the List are replicated, one per task: Objects on screens can initiate run automations and initiate actions. For example, the ADD TASK button navigates to the Add Task screen: The Add Task screen prompts for the values that specify the new task, and the ADD button uses an automation that writes the values to the Tasks table: Automations can be triggered in four different ways. Here’s the automation that generates reminders for tasks that have not been marked as done. The automation runs once for each row in the Tasks table: The notification runs only if the task has not been marked as done, and could also use the FILTER() function: While I don’t have the space to show you how to build an app from scratch, here’s a quick overview. Click Create workbook and Import CSV file or Start from scratch: Click the Tables icon and create reference and data tables: Click the Apps icon and build the app. You can select a wizard that uses your tables as a starting point, or you can start from an empty canvas. Click the Automations icon and add time-driven or data-driven automations: Share the app, as detailed in the next section. Sharing Apps After my app is ready to go, I can share it with other members of my team. Each Honeycode user can be a member of one or more teams: To share my app, I click Share app: Then I search for the desired team members and share the app: They will receive an email that contains a link, and can start using the app immediately. Users with mobile devices can install the Honeycode Player (iOS, Android) and make use of any apps that have been shared with them. Here’s the Simple To-do app: Amazon Honeycode APIs External applications can also use the Honeycode APIs to interact with the applications you build with Honeycode. The functions include: GetScreenData – Retrieve data from any screen of a Honeycode application. InvokeScreenAutomation – Invoke an automation or action defined in the screen of a Honeycode application. Check it Out As you can see, Amazon Honeycode is easy to use, with plenty of power to let you build apps that help you and your team to be more productive. Check it out, build something cool, and let me know what you think! — Jeff; PS – The Amazon Honeycode Forum is the place for you to ask questions, learn from other users, and to find tutorials and other resources that will help you to get started.

AWS Solutions Constructs – A Library of Architecture Patterns for the AWS CDK

Cloud applications are built using multiple components, such as virtual servers, containers, serverless functions, storage buckets, and databases. Being able to provision and configure these resources in a safe, repeatable way is incredibly important to automate your processes and let you focus on the unique parts of your implementation. With the AWS Cloud Development Kit, you can leverage the expressive power of your favorite programming languages to model your applications. You can use high-level components called constructs, preconfigured with “sensible defaults” that you can customize, to quickly build a new application. The CDK provisions your resources using AWS CloudFormation to get all the benefits of managing your infrastructure as code. One of the reasons I like the CDK, is that you can compose and share your own custom components as higher-level constructs. As you can imagine, there are recurring patterns that can be useful to more than one customer. For this reason, today we are launching the AWS Solutions Constructs, an open source extension library for the CDK that provides well-architected patterns to help you build your unique solutions. CDK constructs mostly cover single services. AWS Solutions Constructs provide multi-service patterns that combine two or more CDK resources, and implement best practices such as logging and encryption. Using AWS Solutions Constructs To see the power of a pattern-based approach, let’s take a look at how that works when building a new application. As an example, I want to build an HTTP API to store data in a Amazon DynamoDB table. To keep the content of the table small, I can use DynamoDB Time to Live (TTL) to expire items after a few days. After the TTL expires, data is deleted from the table and sent, via DynamoDB Streams, to a AWS Lambda function to archive the expired data on Amazon Simple Storage Service (S3). To build this application, I can use a few components: An Amazon API Gateway endpoint for the API. A DynamoDB table to store data. A Lambda function to process the API requests, and store data in the DynamoDB table. DynamoDB Streams to capture data changes. A Lambda function processing data changes to archive the expired data. Can I make it simpler? Looking at the available patterns in the AWS Solutions Constructs, I find two that can help me build my app: aws-apigateway-lambda, a Construct that implements an API Gateway REST API connected to a Lambda function. As an example of the “sensible defaults” used by AWS Solutions Constructs, this pattern enables CloudWatch logging for the API Gateway. aws-dynamodb-stream-lambda, a Construct implementing a DynamoDB table streaming data changes to a Lambda function with the least privileged permissions. To build the final architecture, I simply connect those two Constructs together: I am using TypeScript to define the CDK stack, and Node.js for the Lambda functions. Let’s start with the CDK stack:   import * as cdk from '@aws-cdk/core'; import * as lambda from '@aws-cdk/aws-lambda'; import * as apigw from '@aws-cdk/aws-apigateway'; import * as dynamodb from '@aws-cdk/aws-dynamodb'; import { ApiGatewayToLambda } from '@aws-solutions-constructs/aws-apigateway-lambda'; import { DynamoDBStreamToLambda } from '@aws-solutions-constructs/aws-dynamodb-stream-lambda'; export class DemoConstructsStack extends cdk.Stack { constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) { super(scope, id, props); const apiGatewayToLambda = new ApiGatewayToLambda(this, 'ApiGatewayToLambda', { deployLambda: true, lambdaFunctionProps: { code: lambda.Code.fromAsset('lambda'), runtime: lambda.Runtime.NODEJS_12_X, handler: 'restApi.handler' }, apiGatewayProps: { defaultMethodOptions: { authorizationType: apigw.AuthorizationType.NONE } } }); const dynamoDBStreamToLambda = new DynamoDBStreamToLambda(this, 'DynamoDBStreamToLambda', { deployLambda: true, lambdaFunctionProps: { code: lambda.Code.fromAsset('lambda'), runtime: lambda.Runtime.NODEJS_12_X, handler: 'processStream.handler' }, dynamoTableProps: { tableName: 'my-table', partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING }, timeToLiveAttribute: 'ttl' } }); const apiFunction = apiGatewayToLambda.lambdaFunction; const dynamoTable = dynamoDBStreamToLambda.dynamoTable; dynamoTable.grantReadWriteData(apiFunction); apiFunction.addEnvironment('TABLE_NAME', dynamoTable.tableName); } } At the beginning of the stack, I import the standard CDK constructs for the Lambda function, the API Gateway endpoint, and the DynamoDB table. Then, I add the two patterns from the AWS Solutions Constructs, ApiGatewayToLambda and DynamoDBStreamToLambda. After declaring the two ApiGatewayToLambda and DynamoDBStreamToLambda constructs, I store the Lambda function, created by the ApiGatewayToLambda constructs, and the DynamoDB table, created by DynamoDBStreamToLambda, in two variables. At the end of the stack, I “connect” the two patterns together by granting permissions to the Lambda function to read/write in the DynamoDB table, and add the name of the DynamoDB table to the environment of the Lambda function, so that it can be used in the function code to store data in the table. The code of the two Lambda functions is in the lambda folder of the CDK application. I am using the Node.js 12 runtime. The restApi.js function implements the API and writes data to the DynamoDB table. The URL path is used as partition key, all the query string parameters in the URL are stored as attributes. The TTL for the item is computed adding a time window of 7 days to the current time. const { DynamoDB } = require("aws-sdk"); const docClient = new DynamoDB.DocumentClient(); const TABLE_NAME = process.env.TABLE_NAME; const TTL_WINDOW = 7 * 24 * 60 * 60; // 7 days expressed in seconds exports.handler = async function (event) { const item = event.queryStringParameters; item.id = event.pathParameters.proxy; const now = new Date(); item.ttl = Math.round(now.getTime() / 1000) + TTL_WINDOW; const response = await docClient.put({ TableName: TABLE_NAME, Item: item }).promise(); let statusCode = 204; if (response.err != null) { console.error('request: ', JSON.stringify(event, undefined, 2)); console.error('error: ', response.err); statusCode = 500 } return { statusCode: statusCode }; }; The processStream.js function is processing data capture records from the DynamoDB Stream, looking for the items deleted by TTL. The archive functionality is not implemented in this sample code. exports.handler = async function (event) { event.Records.forEach((record) => { console.log('Stream record: ', JSON.stringify(record, null, 2)); if (record.userIdentity.type == "Service" && record.userIdentity.principalId == "dynamodb.amazonaws.com") { // Record deleted by DynamoDB Time to Live (TTL) // I can archive the record to S3, for example using Kinesis Data Firehose. } } }; Let’s see if this works! First, I need to install all dependencies. To simplify dependencies, each release of AWS Solutions Constructs is linked to the corresponding version of the CDK. I this case, I am using version 1.46.0 for both the CDK and the AWS Solutions Constructs patterns. The first three commands are installing plain CDK constructs. The last two commands are installing the AWS Solutions Constructs patterns I am using for this application. npm install @aws-cdk/aws-lambda@1.46.0 npm install @aws-cdk/aws-apigateway@1.46.0 npm install @aws-cdk/aws-dynamodb@1.46.0 npm install @aws-solutions-constructs/aws-apigateway-lambda@1.46.0 npm install @aws-solutions-constructs/aws-dynamodb-stream-lambda@1.46.0 Now, I build the application and use the CDK to deploy the application. npm run build cdk deploy Towards the end of the output of the cdk deploy command, a green light is telling me that the deployment of the stack is completed. Just next, in the Outputs, I find the endpoint of the API Gateway. DemoConstructsStack Outputs: DemoConstructsStack.ApiGatewayToLambdaLambdaRestApiEndpoint9800D4B5 = https://1a2c3c4d.execute-api.eu-west-1.amazonaws.com/prod/ I can now use curl to test the API: curl "https://1a2c3c4d.execute-api.eu-west-1.amazonaws.com/prod/danilop?name=Danilo&amp;company=AWS" Let’s have a look at the DynamoDB table: The item is stored, and the TTL is set. After a week, the item will be deleted and sent via DynamoDB Streams to the processStream.js function. After I complete my testing, I use the CDK again to quickly delete all resources created for this application: cdk destroy Available Now The AWS Solutions Constructs are available now for TypeScript and Python. The AWS Solutions Builders team is working to make these constructs also available when using Java and C# with the CDK, stay tuned. There is no cost in using the AWS Solutions Constructs, or the CDK, you only pay for the resources created when deploying the stack. In this first release, 25 patterns are included, covering lots of different use cases. Which new patterns and features should we focus now? Give use your feedback in the open source project repository! — Danilo

Introducing AWS Snowcone – A Small, Lightweight, Rugged, Secure Edge Computing, Edge Storage, and Data Transfer Device

Last month I published my AWS Snowball Edge Update and told you about the latest updates to Snowball Edge, including faster storage-optimized devices with more memory & vCPUs, the AWS OpsHub for Snow Family GUI-based management tool, IAM for Snowball Edge, and Snowball Edge Support for AWS Systems Manager. AWS Snowcone Today I would like to introduce you to the newest and smallest member of the AWS Snow Family of physical edge computing, edge storage, and data transfer devices for rugged or disconnected environments, AWS Snowcone: AWS Snowcone weighs 4.5 pounds and includes 8 terabytes of usable storage. It is small (9″ long, 6″ wide, and 3″ tall) and rugged, and can be used in a variety of environments including desktops, data centers, messenger bags, vehicles, and in conjunction with drones. Snowcone runs on either AC power or an optional battery, making it great for many different types of use cases where self-sufficiency is vital. The device enclosure is both tamper-evident and tamper-resistant, and also uses a Trusted Platform Module (TPM) designed to ensure both security and full chain-of-custody for your data. The device encrypts data at rest and in transit using keys that are managed by AWS Key Management Service (KMS) and are never stored on the device. Like other Snow Family devices, Snowcone includes an E Ink shipping label designed to ensure the device is automatically sent to the correct AWS facility and to aid in tracking. It also includes 2 CPUs, 4 GB of memory, wired or wireless access, and USB-C power using a cord or the optional battery. There’s enough compute power for you to launch EC2 instances and to use AWS IoT Greengrass. You can use Snowcone for data migration, content distribution, tactical edge computing, healthcare IoT, industrial IoT, transportation, logistics, and autonomous vehicle use cases. You can ship data-laden devices to AWS for offline data transfer, or you can use AWS DataSync for online data transfer. Ordering a Snowcone The ordering process for Snowcone is similar to that for Snowball Edge. I open the Snow Family Console, and click Create Job: I select the Import into Amazon S3 job type and click Next: I choose my address (or enter a new one), and a shipping speed: Next, I give my job a name (Snowcone2) and indicate that I want a Snowcone. I also acknowledge that I will provide my own power supply: Deeper into the page, I choose an S3 bucket for my data, opt-in to WiFi connectivity, and choose an EC2 AMI that will be loaded on the device before it is shipped to me: As you can see from the image, I can choose multiple buckets and/or multiple AMIs. The AMIs must be made from an instance launched from a CentOS or Ubuntu product in AWS Marketplace, and it must contain a SSH key. On successive pages (not shown), I specify permissions (an IAM role), choose an AWS Key Management Service (KMS) key to encrypt my data, and set up a SNS topic for job notifications. Then I confirm my choices and click Create job: Then I await delivery of my device! I can check the status at any time: As noted in one of the earlier screens, I will also need a suitable power supply or battery (you can find several on the Snowcone Accessories page). Time passes, the doorbell rings, Luna barks, and my device is delivered… The console also updates to show that my device has been delivered: On that page, I click Get credentials, copy the client unlock code, and download the manifest file: Setting up My Snowcone I connect the Snowcone to the power supply and to my network, and power up! After a few seconds of initialization, the device shows its IP address and invites me to connect: The IP address was supplied by the DHCP server on my home network, and should be fine. If not, I can touch Network and configure a static IP address or log in to my WiFi network. Next, I download AWS OpsHub for Snow Family, install it, and then configure it to access the device. I select Snowcone and click Next: I enter the IP address as shown on the display: Then I enter the unlock code, upload the manifest file, and click Unlock device: After a minute or two, the device is unlocked and ready. I enter a name (Snowcone1) that I’ll use within AWS OpsHub and click Save profile name: I’m all set: AWS OpsHub for Snow Family Now that I have ordered & received my device, installed AWS OpsHub for Snow Family, and unlocked my device, I am ready to start managing some file storage and doing some edge computing! I click on Get started within Manage file storage, and Start NFS. I have several network options, and I’ll use the defaults: The NFS server is ready within a minute or so, and it has its own IP address: Once it is ready I can mount the NFS volume and copy files to the Snowcone: I can store process these files locally, or I can use AWS DataSync to transfer them to the cloud. As I showed you earlier in this post, I selected an EC2 AMI when I created my job. I can launch instances on the Snowcone using this AMI. I click on Compute, and Launch instance: I have three instance types to choose from: Instance Name CPUs RAM snc1.micro 1 1 GiB snc1.small 1 2 GiB snc1.medium 2 4 GiB I select my AMI & instance type, confirm the networking options, and click Launch: I can also create storage volumes and attach them to the instance. The ability to build AMIs and run them on Snowcones gives you the power to build applications that do all sorts of interesting filtering, pre-processing, and analysis at the edge. I can use AWS DataSync to transfer data from the device to a wide variety of AWS storage services including Amazon Simple Storage Service (S3), Amazon Elastic File System (EFS), or Amazon FSx for Windows File Server. I click on Get started, then Start DataSync Agent, confirm my network settings, and click Start agent: Once the agent is up and running, I copy the IP address: Then I follow the link and create a DataSync agent (the deploy step is not necessary because the agent is already running). I choose an endpoint and paste the IP address of the agent, then click Get key: I give my agent a name (SnowAgent), tag it, and click Create agent: Then I configure the NFS server in the Snowcone as a DataSync location, and use it to transfer data in or out using a DataSync Task. API / CLI While AWS OpsHub is going to be the primary access method for most users, the device can also be accessed programmatically. I can use the Snow Family tools to retrieve the AWS Access Key and Secret Key from the device, create a CLI profile (region is snow), and run commands (or issue API calls) as usual: C:\>aws ec2 \ --endpoint http://192.168.7.154:8008 describe-images \ --profile snowcone1 { "Images": [ { "ImageId": "s.ami-0581034c71faf08d9", "Public": false, "State": "AVAILABLE", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda1", "Ebs": { "DeleteOnTermination": false, "Iops": 0, "SnapshotId": "s.snap-01f2a33baebb50f0e", "VolumeSize": 8 } } ], "Description": "Image for Snowcone delivery #1", "EnaSupport": false, "Name": "Snowcone v1", "RootDeviceName": "/dev/sda1" }, { "ImageId": "s.ami-0bb6828757f6a23cf", "Public": false, "State": "AVAILABLE", "BlockDeviceMappings": [ { "DeviceName": "/dev/sda", "Ebs": { "DeleteOnTermination": true, "Iops": 0, "SnapshotId": "s.snap-003d9042a046f11f9", "VolumeSize": 20 } } ], "Description": "AWS DataSync AMI for online data transfer", "EnaSupport": false, "Name": "scn-datasync-ami", "RootDeviceName": "/dev/sda" } ] } Get One Today You can order a Snowcone today for use in US locations, including here in Seattle: — Jeff;

New – A Shared File System for Your Lambda Functions

I am very happy to announce that AWS Lambda functions can now mount an Amazon Elastic File System (EFS), a scalable and elastic NFS file system storing data within and across multiple availability zones (AZ) for high availability and durability. In this way, you can use a familiar file system interface to store and share data across all concurrent execution environments of one, or more, Lambda functions. EFS supports full file system access semantics, such as strong consistency and file locking. To connect an EFS file system with a Lambda function, you use an EFS access point, an application-specific entry point into an EFS file system that includes the operating system user and group to use when accessing the file system, file system permissions, and can limit access to a specific path in the file system. This helps keeping file system configuration decoupled from the application code. You can access the same EFS file system from multiple functions, using the same or different access points. For example, using different EFS access points, each Lambda function can access different paths in a file system, or use different file system permissions. You can share the same EFS file system with Amazon Elastic Compute Cloud (EC2) instances, containerized applications using Amazon ECS and AWS Fargate, and on-premises servers. Following this approach, you can use different computing architectures (functions, containers, virtual servers) to process the same files. For example, a Lambda function reacting to an event can update a configuration file that is read by an application running on containers. Or you can use a Lambda function to process files uploaded by a web application running on EC2. In this way, some use cases are much easier to implement with Lambda functions. For example: Processing or loading data larger than the space available in /tmp (512MB). Loading the most updated version of files that change frequently. Using data science packages that require storage space to load models and other dependencies. Saving function state across invocations (using unique file names, or file system locks). Building applications requiring access to large amounts of reference data. Migrating legacy applications to serverless architectures. Interacting with data intensive workloads designed for file system access. Partially updating files (using file system locks for concurrent access). Moving a directory and all its content within a file system with an atomic operation. Creating an EFS File System To mount an EFS file system, your Lambda functions must be connected to an Amazon Virtual Private Cloud that can reach the EFS mount targets. For simplicity, I am using here the default VPC that is automatically created in each AWS Region. Note that, when connecting Lambda functions to a VPC, networking works differently. If your Lambda functions are using Amazon Simple Storage Service (S3) or Amazon DynamoDB, you should create a gateway VPC endpoint for those services. If your Lambda functions need to access the public internet, for example to call an external API, you need to configure a NAT Gateway. I usually don’t change the configuration of my default VPCs. If I have specific requirements, I create a new VPC with private and public subnets using the AWS Cloud Development Kit, or use one of these AWS CloudFormation sample templates. In this way, I can manage networking as code. In the EFS console, I select Create file system and make sure that the default VPC and its subnets are selected. For all subnets, I use the default security group that gives network access to other resources in the VPC using the same security group. In the next step, I give the file system a Name tag and leave all other options to their default values. Then, I select Add access point. I use 1001 for the user and group IDs and limit access to the /message path. In the Owner section, used to create the folder automatically when first connecting to the access point, I use the same user and group IDs as before, and 750 for permissions. With this permissions, the owner can read, write, and execute files. Users in the same group can only read. Other users have no access. I go on, and complete the creation of the file system. Using EFS with Lambda Functions To start with a simple use case, let’s build a Lambda function implementing a MessageWall API to add, read, or delete text messages. Messages are stored in a file on EFS so that all concurrent execution environments of that Lambda function see the same content. In the Lambda console, I create a new MessageWall function and select the Python 3.8 runtime. In the Permissions section, I leave the default. This will create a new AWS Identity and Access Management (IAM) role with basic permissions. When the function is created, in the Permissions tab I click on the IAM role name to open the role in the IAM console. Here, I select Attach policies to add the AWSLambdaVPCAccessExecutionRole and AmazonElasticFileSystemClientReadWriteAccess AWS managed policies. In a production environment, you can restrict access to a specific VPC and EFS access point. Back in the Lambda console, I edit the VPC configuration to connect the MessageWall function to all subnets in the default VPC, using the same default security group I used for the EFS mount points. Now, I select Add file system in the new File system section of the function configuration. Here, I choose the EFS file system and accesss point I created before. For the local mount point, I use /mnt/msg and Save. This is the path where the access point will be mounted, and corresponds to the /message folder in my EFS file system. In the Function code editor of the Lambda console, I paste the following code and Save. import os import fcntl MSG_FILE_PATH = '/mnt/msg/content' def get_messages(): try: with open(MSG_FILE_PATH, 'r') as msg_file: fcntl.flock(msg_file, fcntl.LOCK_SH) messages = msg_file.read() fcntl.flock(msg_file, fcntl.LOCK_UN) except: messages = 'No message yet.' return messages def add_message(new_message): with open(MSG_FILE_PATH, 'a') as msg_file: fcntl.flock(msg_file, fcntl.LOCK_EX) msg_file.write(new_message + "\n") fcntl.flock(msg_file, fcntl.LOCK_UN) def delete_messages(): try: os.remove(MSG_FILE_PATH) except: pass def lambda_handler(event, context): method = event['requestContext']['http']['method'] if method == 'GET': messages = get_messages() elif method == 'POST': new_message = event['body'] add_message(new_message) messages = get_messages() elif method == 'DELETE': delete_messages() messages = 'Messages deleted.' else: messages = 'Method unsupported.' return messages I select Add trigger and in the configuration I select the Amazon API Gateway. I create a new HTTP API. For simplicity, I leave my API endpoint open. With the API Gateway trigger selected, I copy the endpoint of the new API I just created. I can now use curl to test the API: $ curl https://1a2b3c4d5e.execute-api.us-east-1.amazonaws.com/default/MessageWall No message yet. $ curl -X POST -H "Content-Type: text/plain" -d 'Hello from EFS!' https://1a2b3c4d5e.execute-api.us-east-1.amazonaws.com/default/MessageWall Hello from EFS! $ curl -X POST -H "Content-Type: text/plain" -d 'Hello again :)' https://1a2b3c4d5e.execute-api.us-east-1.amazonaws.com/default/MessageWall Hello from EFS! Hello again :) $ curl https://1a2b3c4d5e.execute-api.us-east-1.amazonaws.com/default/MessageWall Hello from EFS! Hello again :) $ curl -X DELETE https://1a2b3c4d5e.execute-api.us-east-1.amazonaws.com/default/MessageWall Messages deleted. $ curl https://1a2b3c4d5e.execute-api.us-east-1.amazonaws.com/default/MessageWall No message yet. It would be relatively easy to add unique file names (or specific subdirectories) for different users and extend this simple example into a more complete messaging application. As a developer, I appreciate the simplicity of using a familiar file system interface in my code. However, depending on your requirements, EFS throughput configuration must be taken into account. See the section Understanding EFS performance later in the post for more information. Now, let’s use the new EFS file system support in AWS Lambda to build something more interesting. For example, let’s use the additional space available with EFS to build a machine learning inference API processing images. Building a Serverless Machine Learning Inference API To create a Lambda function implementing machine learning inference, I need to be able, in my code, to import the necessary libraries and load the machine learning model. Often, when doing so, the overall size of those dependencies goes beyond the current AWS Lambda limits in the deployment package size. One way of solving this is to accurately minimize the libraries to ship with the function code, and then download the model from an S3 bucket straight to memory (up to 3 GB, including the memory required for processing the model) or to /tmp (up 512 MB). This custom minimization and download of the model has never been easy to implement. Now, I can use an EFS file system. The Lambda function I am building this time needs access to the public internet to download a pre-trained model and the images to run inference on. So I create a new VPC with public and private subnets, and configure a NAT Gateway and the route table used by the the private subnets to give access to the public internet. Using the AWS Cloud Development Kit, it’s just a few lines of code. I create a new EFS file system and an access point in the new VPC using similar configurations as before. This time, I use /ml for the access point path. Then, I create a new MLInference Lambda function with the same set up as before for permissions and connect the function to the private subnets of the new VPC. Machine learning inference is quite a heavy workload, so I select 3 GB for memory and 5 minutes for timeout. In the File system configuration, I add the new access point and mount it under /mnt/inference. The machine learning framework I am using for this function is PyTorch, and I need to put the libraries required to run inference in the EFS file system. I launch an Amazon Linux EC2 instance in a public subnet of the new VPC. In the instance details, I select one of the availability zones where I have an EFS mount point, and then Add file system to automatically mount the same EFS file system I am using for the function. For the security groups of the EC2 instance, I select the default security group (to be able to mount the EFS file system) and one that gives inbound access to SSH (to be able to connect to the instance). I connect to the instance using SSH and create a requirements.txt file containing the dependencies I need: torch torchvision numpy The EFS file system is automatically mounted by EC2 under /mnt/efs/fs1. There, I create the /ml directory and change the owner of the path to the user and group I am using now that I am connected (ec2-user). $ sudo mkdir /mnt/efs/fs1/ml $ sudo chown ec2-user:ec2-user /mnt/efs/fs1/ml I install Python 3 and use pip to install the dependencies in the /mnt/efs/fs1/ml/lib path: $ sudo yum install python3 $ pip3 install -t /mnt/efs/fs1/ml/lib -r requirements.txt Finally, I give ownership of the whole /ml path to the user and group I used for the EFS access point: $ sudo chown -R 1001:1001 /mnt/efs/fs1/ml Overall, the dependencies in my EFS file system are using about 1.5 GB of storage. I go back to the MLInference Lambda function configuration. Depending on the runtime you use, you need to find a way to tell where to look for dependencies if they are not included with the deployment package or in a layer. In the case of Python, I set the PYTHONPATH environment variable to /mnt/inference/lib. I am going to use PyTorch Hub to download this pre-trained machine learning model to recognize the kind of bird in a picture. The model I am using for this example is relatively small, about 200 MB. To cache the model on the EFS file system, I set the TORCH_HOME environment variable to /mnt/inference/model. All dependencies are now in the file system mounted by the function, and I can type my code straight in the Function code editor. I paste the following code to have a machine learning inference API: import urllib import json import os import torch from PIL import Image from torchvision import transforms transform_test = transforms.Compose([ transforms.Resize((600, 600), Image.BILINEAR), transforms.CenterCrop((448, 448)), transforms.ToTensor(), transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)), ]) model = torch.hub.load('nicolalandro/ntsnet-cub200', 'ntsnet', pretrained=True, **{'topN': 6, 'device': 'cpu', 'num_classes': 200}) model.eval() def lambda_handler(event, context): url = event['queryStringParameters']['url'] img = Image.open(urllib.request.urlopen(url)) scaled_img = transform_test(img) torch_images = scaled_img.unsqueeze(0) with torch.no_grad(): top_n_coordinates, concat_out, raw_logits, concat_logits, part_logits, top_n_index, top_n_prob = model(torch_images) _, predict = torch.max(concat_logits, 1) pred_id = predict.item() bird_class = model.bird_classes[pred_id] print('bird_class:', bird_class) return json.dumps({ "bird_class": bird_class, }) I add the API Gateway as trigger, similarly to what I did before for the MessageWall function. Now, I can use the serverless API I just created to analyze pictures of birds. I am not really an expert in the field, so I looked for a couple of interesting images on Wikipedia: An atlantic puffin. A western grebe. I call the API to get a prediction for these two pictures: $ curl https://1a2b3c4d5e.execute-api.us-east-1.amazonaws.com/default/MLInference?url=https://path/to/image/atlantic-puffin.jpg {"bird_class": "106.Horned_Puffin"} $ curl https://1a2b3c4d5e.execute-api.us-east-1.amazonaws.com/default/MLInference?url=https://path/to/image/western-grebe.jpg {"bird_class": "053.Western_Grebe"} It works! Looking at Amazon CloudWatch Logs for the Lambda function, I see that the first invocation, when the function loads and prepares the pre-trained model for inference on CPUs, takes about 30 seconds. To avoid a slow response, or a timeout from the API Gateway, I use Provisioned Concurrency to keep the function ready. The next invocations take about 1.8 seconds. Understanding EFS Performance When using EFS with your Lambda function, is very important to understand how EFS performance work. For throughput, each file system can be configured to use bursting or provisioned mode. When using bursting mode, all EFS file systems, regardless of size, can burst at least to 100 MiB/s of throughput. Those over 1 TiB in the standard storage class can burst to 100 MiB/s per TiB of data stored in the file system. EFS uses a credit system to determine when file systems can burst. Each file system earns credits over time at a baseline rate that is determined by the size of the file system that is stored in the standard storage class. A file system uses credits whenever it reads or writes data. The baseline rate is 50 KiB/s per GiB of storage. You can monitor the use of credits in CloudWatch, each EFS file system has a BurstCreditBalance metric. If you see that you are consuming all credits, and the BurstCreditBalance metric is going to zero, you should enable provisioned throughput mode for the file system, from 1 to 1024 MiB/s. There is an additional cost when using provisioned throughput, based on how much throughput you are adding on top of the baseline rate. To avoid running out of credits, you should think of the throughput as the average you need during the day. For example, if you have a 10GB file system, you have 500 KiB/s of baseline rate, and every day you can read/write 500 KiB/s * 3600 seconds * 24 hours = 43.2 GiB. If the libraries and everything you function needs to load during initialization are about 2 GiB, and you only access the EFS file system during function initialization, like in the MLInference Lambda function above, that means you can initialize your function (for example because of updates or scaling up activities) about 20 times per day. That’s not a lot, and you would probably need to configure provisioned throughput for the EFS file system. If you have 10 MiB/s of provisioned throughput, then every day you have 10 MiB/s * 3600 seconds * 24 hours = 864 GiB to read or write. If you only use the EFS file system at function initialization to read about 2 GB of dependencies, it means that you can have 400 initializations per day. That may be enough for your use case. In the Lambda function configuration, you can also use the reserve concurrency control to limit the maximum number of execution environments used by a function. If, by mistake, the BurstCreditBalance goes down to zero, and the file system is relatively small (for example, a few GiBs), there is the possibility that your function gets stuck and can’t execute fast enough before reaching the timeout. In that case, you should enable (or increase) provisioned throughput for the EFS file system, or throttle your function by setting the reserved concurrency to zero to avoid all invocations until the EFS file system has enough credits. Understanding Security Controls When using EFS file systems with AWS Lambda, you have multiple levels of security controls. I’m doing a quick recap here because they should all be considered during the design and implementation of your serverless applications. You can find more info on using IAM authorization and access points with EFS in this post. To connect a Lambda function to an EFS file system, you need: Network visibility in terms of VPC routing/peering and security group. IAM permissions for the Lambda function to access the VPC and mount (read only or read/write) the EFS file system. You can specify in the IAM policy conditions which EFS access point the Lambda function can use. The EFS access point can limit access to a specific path in the file system. File system security (user ID, group ID, permissions) can limit read, write, or executable access for each file or directory mounted by a Lambda function. The Lambda function execution environment and the EFS mount point uses industry standard Transport Layer Security (TLS) 1.2 to encrypt data in transit. You can provision Amazon EFS to encrypt data at rest. Data encrypted at rest is transparently encrypted while being written, and transparently decrypted while being read, so you don’t have to modify your applications. Encryption keys are managed by the AWS Key Management Service (KMS), eliminating the need to build and maintain a secure key management infrastructure. Available Now This new feature is offered in all regions where AWS Lambda and Amazon EFS are available, with the exception of the regions in China, where we are working to make this integration available as soon as possible. For more information on availability, please see the AWS Region table. To learn more, please see the documentation. EFS for Lambda can be configured using the console, the AWS Command Line Interface (CLI), the AWS SDKs, and the Serverless Application Model. This feature allows you to build data intensive applications that need to process large files. For example, you can now unzip a 1.5 GB file in a few lines of code, or process a 10 GB JSON document. You can also load libraries or packages that are larger than the 250 MB package deployment size limit of AWS Lambda, enabling new machine learning, data modelling, financial analysis, and ETL jobs scenarios. Amazon EFS for Lambda is supported at launch in AWS Partner Network solutions, including Epsagon, Lumigo, Datadog, HashiCorp Terraform, and Pulumi. There is no additional charge for using EFS from Lambda functions. You pay the standard price for AWS Lambda and Amazon EFS. Lambda execution environments always connect to the right mount target in an AZ and not across AZs. You can connect to EFS in the same AZ via cross account VPC but there can be data transfer costs for that. We do not support cross region, or cross AZ connectivity between EFS and Lambda. — Danilo

Amazon EKS Now Supports EC2 Inf1 Instances

Amazon Elastic Kubernetes Service (EKS) has quickly become a leading choice for machine learning workloads. It combines the developer agility and the scalability of Kubernetes, with the wide selection of Amazon Elastic Compute Cloud (EC2) instance types available on AWS, such as the C5, P3, and G4 families. As models become more sophisticated, hardware acceleration is increasingly required to deliver fast predictions at high throughput. Today, we’re very happy to announce that AWS customers can now use the Amazon EC2 Inf1 instances on Amazon Elastic Kubernetes Service, for high performance and the lowest prediction cost in the cloud. A primer on EC2 Inf1 instances Inf1 instances were launched at AWS re:Invent 2019. They are powered by AWS Inferentia, a custom chip built from the ground up by AWS to accelerate machine learning inference workloads. Inf1 instances are available in multiple sizes, with 1, 4, or 16 AWS Inferentia chips, with up to 100 Gbps network bandwidth and up to 19 Gbps EBS bandwidth. An AWS Inferentia chip contains four NeuronCores. Each one implements a high-performance systolic array matrix multiply engine, which massively speeds up typical deep learning operations such as convolution and transformers. NeuronCores are also equipped with a large on-chip cache, which helps cut down on external memory accesses, saving I/O time in the process. When several AWS Inferentia chips are available on an Inf1 instance, you can partition a model across them and store it entirely in cache memory. Alternatively, to serve multi-model predictions from a single Inf1 instance, you can partition the NeuronCores of an AWS Inferentia chip across several models. Compiling Models for EC2 Inf1 Instances To run machine learning models on Inf1 instances, you need to compile them to a hardware-optimized representation using the AWS Neuron SDK. All tools are readily available on the AWS Deep Learning AMI, and you can also install them on your own instances. You’ll find instructions in the Deep Learning AMI documentation, as well as tutorials for TensorFlow, PyTorch, and Apache MXNet in the AWS Neuron SDK repository. In the demo below, I will show you how to deploy a Neuron-optimized model on an EKS cluster of Inf1 instances, and how to serve predictions with TensorFlow Serving. The model in question is BERT, a state of the art model for natural language processing tasks. This is a huge model with hundreds of millions of parameters, making it a great candidate for hardware acceleration. Building an EKS Cluster of EC2 Inf1 Instances First of all, let’s build a cluster with two inf1.2xlarge instances. I can easily do this with eksctl, the command-line tool to provision and manage EKS clusters. You can find installation instructions in the EKS documentation. Here is the configuration file for my cluster. Eksctl detects that I’m launching a node group with an Inf1 instance type, and will start your worker nodes using the EKS-optimized Accelerated AMI. apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: cluster-inf1 region: us-west-2 nodeGroups: - name: ng1-public instanceType: inf1.2xlarge minSize: 0 maxSize: 3 desiredCapacity: 2 ssh: allow: true Then, I use eksctl to create the cluster. This process will take approximately 10 minutes. $ eksctl create cluster -f inf1-cluster.yaml Eksctl automatically installs the Neuron device plugin in your cluster. This plugin advertises Neuron devices to the Kubernetes scheduler, which can be requested by containers in a deployment spec. I can check with kubectl that the device plug-in container is running fine on both Inf1 instances. $ kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE aws-node-tl5xv 1/1 Running 0 14h aws-node-wk6qm 1/1 Running 0 14h coredns-86d5cbb4bd-4fxrh 1/1 Running 0 14h coredns-86d5cbb4bd-sts7g 1/1 Running 0 14h kube-proxy-7px8d 1/1 Running 0 14h kube-proxy-zqvtc 1/1 Running 0 14h neuron-device-plugin-daemonset-888j4 1/1 Running 0 14h neuron-device-plugin-daemonset-tq9kc 1/1 Running 0 14h Next, I define AWS credentials in a Kubernetes secret. They will allow me to grab my BERT model stored in S3. Please note that both keys needs to be base64-encoded. apiVersion: v1 kind: Secret metadata: name: aws-s3-secret type: Opaque data: AWS_ACCESS_KEY_ID: <base64-encoded value> AWS_SECRET_ACCESS_KEY: <base64-encoded value> Finally, I store these credentials on the cluster. $ kubectl apply -f secret.yaml The cluster is correctly set up. Now, let’s build an application container storing a Neuron-enabled version of TensorFlow Serving. Building an Application Container for TensorFlow Serving The Dockerfile is very simple. We start from an Amazon Linux 2 base image. Then, we install the AWS CLI, and the TensorFlow Serving package available in the Neuron repository. FROM amazonlinux:2 RUN yum install -y awscli RUN echo $'[neuron] \n\ name=Neuron YUM Repository \n\ baseurl=https://yum.repos.neuron.amazonaws.com \n\ enabled=1' > /etc/yum.repos.d/neuron.repo RUN rpm --import https://yum.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB RUN yum install -y tensorflow-model-server-neuron I build the image, create an Amazon Elastic Container Registry repository, and push the image to it. $ docker build . -f Dockerfile -t tensorflow-model-server-neuron $ docker tag IMAGE_NAME 123456789012.dkr.ecr.us-west-2.amazonaws.com/inf1-demo $ aws ecr create-repository --repository-name inf1-demo $ docker push 123456789012.dkr.ecr.us-west-2.amazonaws.com/inf1-demo Our application container is ready. Now, let’s define a Kubernetes service that will use this container to serve BERT predictions. I’m using a model that has already been compiled with the Neuron SDK. You can compile your own using the instructions available in the Neuron SDK repository. Deploying BERT as a Kubernetes Service The deployment manages two containers: the Neuron runtime container, and my application container. The Neuron runtime runs as a sidecar container image, and is used to interact with the AWS Inferentia chips. At startup, the latter configures the AWS CLI with the appropriate security credentials. Then, it fetches the BERT model from S3. Finally, it launches TensorFlow Serving, loading the BERT model and waiting for prediction requests. For this purpose, the HTTP and grpc ports are open. Here is the full manifest. kind: Service apiVersion: v1 metadata: name: eks-neuron-test labels: app: eks-neuron-test spec: ports: - name: http-tf-serving port: 8500 targetPort: 8500 - name: grpc-tf-serving port: 9000 targetPort: 9000 selector: app: eks-neuron-test role: master type: ClusterIP --- kind: Deployment apiVersion: apps/v1 metadata: name: eks-neuron-test labels: app: eks-neuron-test role: master spec: replicas: 2 selector: matchLabels: app: eks-neuron-test role: master template: metadata: labels: app: eks-neuron-test role: master spec: volumes: - name: sock emptyDir: {} containers: - name: eks-neuron-test image: 123456789012.dkr.ecr.us-west-2.amazonaws.com/inf1-demo:latest command: ["/bin/sh","-c"] args: - "mkdir ~/.aws/ && \ echo '[eks-test-profile]' > ~/.aws/credentials && \ echo AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID >> ~/.aws/credentials && \ echo AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY >> ~/.aws/credentials; \ /usr/bin/aws --profile eks-test-profile s3 sync s3://jsimon-inf1-demo/bert /tmp/bert && \ /usr/local/bin/tensorflow_model_server_neuron --port=9000 --rest_api_port=8500 --model_name=bert_mrpc_hc_gelus_b4_l24_0926_02 --model_base_path=/tmp/bert/" ports: - containerPort: 8500 - containerPort: 9000 imagePullPolicy: Always env: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: key: AWS_ACCESS_KEY_ID name: aws-s3-secret - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: key: AWS_SECRET_ACCESS_KEY name: aws-s3-secret - name: NEURON_RTD_ADDRESS value: unix:/sock/neuron.sock resources: limits: cpu: 4 memory: 4Gi requests: cpu: "1" memory: 1Gi volumeMounts: - name: sock mountPath: /sock - name: neuron-rtd image: 790709498068.dkr.ecr.us-west-2.amazonaws.com/neuron-rtd:1.0.6905.0 securityContext: capabilities: add: - SYS_ADMIN - IPC_LOCK volumeMounts: - name: sock mountPath: /sock resources: limits: hugepages-2Mi: 256Mi aws.amazon.com/neuron: 1 requests: memory: 1024Mi I use kubectl to create the service. $ kubectl create -f bert_service.yml A few seconds later, the pods are up and running. $ kubectl get pods NAME                           READY STATUS  RESTARTS AGE eks-neuron-test-5d59b55986-7kdml 2/2   Running 0        14h eks-neuron-test-5d59b55986-gljlq 2/2   Running 0        14h Finally, I redirect service port 9000 to local port 9000, to let my prediction client connect locally. $ kubectl port-forward svc/eks-neuron-test 9000:9000 & Now, everything is ready for prediction, so let’s invoke the model. Predicting with BERT on EKS and Inf1 The inner workings of BERT are beyond the scope of this post. This particular model expects a sequence of 128 tokens, encoding the words of two sentences we’d like to compare for semantic equivalence. Here, I’m only interested in measuring prediction latency, so dummy data is fine. I build 100 prediction requests storing a sequence of 128 zeros. I send them to the TensorFlow Serving endpoint via grpc, and I compute the average prediction time. import numpy as np import grpc import tensorflow as tf from tensorflow_serving.apis import predict_pb2 from tensorflow_serving.apis import prediction_service_pb2_grpc import time if __name__ == '__main__': channel = grpc.insecure_channel('localhost:9000') stub = prediction_service_pb2_grpc.PredictionServiceStub(channel) request = predict_pb2.PredictRequest() request.model_spec.name = 'bert_mrpc_hc_gelus_b4_l24_0926_02' i = np.zeros([1, 128], dtype=np.int32) request.inputs['input_ids'].CopyFrom(tf.contrib.util.make_tensor_proto(i, shape=i.shape)) request.inputs['input_mask'].CopyFrom(tf.contrib.util.make_tensor_proto(i, shape=i.shape)) request.inputs['segment_ids'].CopyFrom(tf.contrib.util.make_tensor_proto(i, shape=i.shape)) latencies = [] for i in range(100): start = time.time() result = stub.Predict(request) latencies.append(time.time() - start) print("Inference successful: {}".format(i)) print ("Ran {} inferences successfully. Latency average = {}".format(len(latencies), np.average(latencies))) On average, prediction took 5.92ms. As far as BERT goes, this is pretty good! Ran 100 inferences successfully. Latency average = 0.05920819044113159 In real-life, we would certainly be batching prediction requests in order to increase throughput. If needed, we could also scale to larger Inf1 instances supporting several Inferentia chips, and deliver even more prediction performance at low cost. Getting Started Kubernetes users can deploy Amazon Elastic Compute Cloud (EC2) Inf1 instances on Amazon Elastic Kubernetes Service today in the US East (N. Virginia) and US West (Oregon) regions. As Inf1 deployment progresses, you’ll be able to use them with Amazon Elastic Kubernetes Service in more regions. Give this a try, and please send us feedback either through your usual AWS Support contacts, on the AWS Forum for Amazon Elastic Kubernetes Service, or on the container roadmap on Github. - Julien

Software Package Management with AWS CodeArtifact

Software artifact repositories and their associated package managers are an essential component of development. Downloading and referencing pre-built libraries of software with a package manager, at the point in time the libraries are needed, simplifies both development and build processes. A variety of package repositories can be used, for example Maven Central, npm public registry, and PyPi (Python Package Index), among others. Working with a multitude of artifact repositories can present some challenges to organizations that want to carefully control both versions of, and access to, the software dependencies of their applications. Any changes to dependencies need to be controlled, to try and prevent undetected and exploitable vulnerabilities creeping into the organization’s applications. By using a centralized repository, it becomes easier for organizations to manage access control and version changes, and gives teams confidence that when updating package versions, the new versions have been approved for use by their IT leaders. Larger organizations may turn to traditional artifact repository software to solve these challenges, but these products can introduce additional challenges around installation, configuration, maintenance, and scaling. For smaller organizations, the price and maintenance effort of traditional artifact repository software may be prohibitive. Generally available today, AWS CodeArtifact is a fully managed artifact repository service for developers and organizations to help securely store and share the software packages used in their development, build, and deployment processes. Today, CodeArtifact can be used with popular build tools and package managers such as Maven and Gradle (for Java), npm and yarn (for Javascript), and pip and twine (for Python), with more to come. As new packages are ingested, or published to your repositories, CodeArtifact automatically scales, and as a fully managed service, CodeArtifact requires no infrastructure installation or maintenance on your part. Additionally, CodeArtifact is a polyglot artifact repository, meaning it can store artifact packages of any supported type. For example, a single CodeArtifact repository could be configured to store packages from Maven, npm and Python repositories side by side in one location. CodeArtifact repositories are organized into a domain. We recommend that you use a single domain for your organization, and then add repositories to it. For example you might choose to use different repositories for different teams. To publish packages into your repositories, or ingest packages from external repositories, you simply use the package manager tools your developers are used to. Let’s take a look at the process of getting started. Getting started with CodeArtifact To get started with CodeArtifact, I first need to create a domain for my organization, which will aggregate my repositories. Domains are used to perform the actual storage of packages and metadata, even though I consume them from a repository. This has the advantage that a single package asset, for example a given npm package, would be stored only once per domain no matter how many repositories it may appear to be in. From the CodeArtifact console, I can select Domains from the left-hand navigation panel, or instead create a domain as part of creating my first repository, which I’ll do here by clicking Create repository. First, I give my repository a name and optional description, and I then have the option to connect my repository to several upstream repositories. When requests are made for packages not present in my repository, CodeArtifact will pull the respective packages from these upstream repositories for me, and cache them into my CodeArtifact repository. Note that a CodeArtifact repository can also act as an upstream for other CodeArtifact repositories. For the example here, I’m going to pull packages from the npm public registry and PyPi. CodeArtifact will refer to the repositories it creates on my behalf to manage these external connections as npm-store and pypi-store. Clicking Next, I then select, or create, a domain which I do by choosing the account that will own the domain and then giving the domain a name. Note that CodeArtifact encrypts all assets and metadata in a domain using a single AWS Key Management Service (KMS) key. Here, I’m going to use a key that will be created for me by the service, but I can elect to use my own. Clicking Next takes me to the final step to review my settings, and I can confirm the package flow from my selected upstream repositories is as I expect. Clicking Create repository completes the process, and in this case creates the domain, my repository, and two additional repositories representing the upstreams. After using this simple setup process, my domain and its initial repository, configured to pull upstream from npm and PyPi, are now ready to hold software artifact packages, and I could also add additional repositories if needed. However my next step for this example is to configure the package managers for my upstream repositories, npm and pip, with access to the CodeArtifact repository, as follows. Configuring package managers The steps to configure various package managers can be found in the documentation, but conveniently the console also gives me the instructions I need when I select my repository. I’m going to start with npm, and I can access the instructions by first selecting my npm-pypi-example-repository and clicking View connection instructions. In the resulting dialog I select the package manager I want to configure and I am shown the relevant instructions. I have the choice of using the AWS Command Line Interface (CLI) to manage the whole process (for npm, pip, and twine), or I can use a CLI command to get the token and then run npm commands to attach the token to the repository reference. Regardless of the package manager, or the set of instructions I follow, the commands simply attach an authorization token, which is valid for 12 hours, to the package manager configuration for the repository. So that I don’t forget to refresh the token, I have taken the approach of adding the relevant command to my startup profile so that my token is automatically refreshed at the start of each day. Following the same guidance, I similarly configure pip, again using the AWS CLI approach: C:\> aws codeartifact login --tool pip --repository npm-pypi-example-repository --domain my-example-domain --domain-owner ACCOUNT_ID Writing to C:\Users\steve\AppData\Roaming\pip\pip.ini Successfully logged in to codeartifact for pypi That’s it! I’m now ready to start using the single repository for dependencies in my Node.js and Python applications. Any dependency I add which is not already in the repository will be fetched from the designated upstream repositories and added to my CodeArtifact repository. Let’s try some simple tests to close out the post. First, after changing to an empty directory, I execute a simple npm install command, in this case to install the AWS Cloud Development Kit. npm install -g aws-cdk Selecting the repository in the CodeArtifact console, I can see that the packages for the AWS Cloud Development Kit, and its dependencies, have now been downloaded from the upstream npm public registry repository, and added to my repository. I mentioned earlier that CodeArtifact repositories are polyglot, and able to store packages of any supported type. Let’s now add a Python package, in this case Pillow, a popular image manipulation library. > pip3 install Pillow Looking in indexes: https://aws:****@my-example-domain-123456789012.d.codeartifact.us-west-2.amazonaws.com/pypi/npm-pypi-example-repository/simple/ Collecting Pillow Downloading https://my-example-domain-123456789012.d.codeartifact.us-west-2.amazonaws.com/pypi/npm-pypi-example-repository/simple/pillow/7.1.2/Pillow-7.1.2-cp38-cp38-win_amd64.whl (2.0 MB) |████████████████████████████████| 2.0 MB 819 kB/s Installing collected packages: Pillow Successfully installed Pillow-7.1.2 In the console, I can see the Python package sitting alongside the npm packages I added earlier. Although I’ve used the console to verify my actions, I could equally well use CLI commands. For example, to list the repository packages I could have run the following command: aws codeartifact list-packages --domain my-example-domain --repository npm-pypi-example-repository As you might expect, additional commands are available to help with work with domains, repositories, and the packages they contain. Availability AWS CodeArtifact is now generally available in the Frankfurt, Ireland, Mumbai, N.Virginia, Ohio, Oregon, Singapore, Sweden, Sydney, and Tokyo regions. AWS CloudFormation support for CodeArtifact is coming soon. For additional best practice considerations on using CodeArtifact, see this blog post, and tune in on June 12th at noon (PST) to Twitch.tv/aws or LinkedIn Live, where we will be showing how you can get started with CodeArtifact. — Steve

New – Label 3D Point Clouds with Amazon SageMaker Ground Truth

Launched at AWS re:Invent 2018, Amazon Sagemaker Ground Truth is a capability of Amazon SageMaker that makes it easy to annotate machine learning datasets. Customers can efficiently and accurately label image and text data with built-in workflows, or any other type of data with custom workflows. Data samples are automatically distributed to a workforce (private, 3rd party or MTurk), and annotations are stored in Amazon Simple Storage Service (S3). Optionally, automated data labeling may also be enabled, reducing both the amount of time required to label the dataset, and the associated costs. About a year ago, I met with Automotive customers who expressed interest in labeling 3-dimensional (3D) datasets for autonomous driving. Captured by LIDAR sensors, these datasets are particularly large and complex. Data is stored in frames that typically contain 50,000 to 5 million points, and can weigh up to hundreds of Megabytes each. Frames are either stored individually, or in sequences that make it easier to track moving objects. As you can imagine, labeling these datasets is extremely time-consuming, as workers need to navigate complex 3D scenes and annotate many different object classes. This often requires building and managing very complex tools. Always looking to help customers build simpler and more efficient workflows, the Ground Truth team gathered more feedback, and got to work. Today, I’m extremely happy to announce that you can use Amazon Sagemaker Ground Truth to label 3D point clouds using a built-in editor, and state-of-the-art assistive labeling features. Introducing 3D Point Cloud Labeling Just like for other Ground Truth tasks types, input data for 3D point clouds has to be stored in an S3 bucket. It also needs to be described by a manifest file, a JSON file containing both the location of frames in S3 and their attributes. A dataset may contain either single-frame data, or multi-frame sequences. Optionally, the dataset may also include image data captured by on-board cameras. Using a feature called “sensor fusion”, Ground Truth can synchronize a 3D point cloud with up to 8 cameras. Thanks to this, workers get a real-life view of the scene, and they can also interchangeably apply labels to 2D images and 3D point clouds. Once the manifest file is ready, Ground Truth lets you create the following task types: Object Detection: identify objects of interest within a 3D point cloud frame. Object Tracking: track objects of interest across a sequence of 3D point cloud frames. Semantic Segmentation: segment the points of a 3D point cloud frame into predefined categories. These can either be labeling jobs where workers annotate new frames, or adjustment jobs where they review and fine-tune existing annotations. Jobs may be distributed either to a private workforce or to a vendor workforce you picked on AWS Marketplace. Using the built-in graphical user interface (GUI) and its shortcuts for navigation and labeling, workers can quickly and accurately apply labels, boxes and categories to 3D objects (“car”, “pedestrian”, and so on). They can also add user-defined attributes, such as the color of a car, or whether an object is fully or partially visible. The GUI includes many assistive labeling features that significantly simplify labeling work, save time, and improve the quality of annotations. Here are a few examples: Snapping: Ground Truth infers a tight-fitting box around the object. Interpolation: the labeler annotates an object in the first and last frames of a sequence. Ground Truth automatically annotates it in the middle frames. Ground detection and removal: Ground Truth can automatically detect and remove 3D points belonging to the ground from object boxes. Even with assistive labeling, it may take a while to annotate complex frames and sequences, so work is saved periodically to avoid any data loss. Preparing 3D Point Cloud Datasets As previously mentioned, you have to provide a manifest file describing your 3D dataset. The format of this file is defined in the Ground Truth documentation. Of course, the steps required to build it will vary from one dataset to the next. For example, the Audi A2D2 dataset contains almost 400,000 frames, with 360-degree 3D LIDAR data and 2D images. KITTI, another popular choice for autonomous driving research, includes a 3D dataset with 15,000 images and their corresponding point clouds, for a total of 80,256 labeled objects. This notebook shows you how to convert KITTI data to the Ground Truth format. When datasets contain both 3D LIDAR data and 2D camera images, one challenge is to synchronize them. This allows us to project 3D points to 2D coordinates, map them on the pictures captured by on-board cameras, and vice versa. Another challenge is that data captured by a given device uses coordinates local to this device. Fortunately, we know where the device is located on the car, and where it’s pointed to. All of this can be solved by building a global coordinate system, also known as a World Coordinate System (WCS). Using matrix operations (which I’ll spare you), we can compute the coordinates of all data points inside the WCS. Once frames have been processed, their information is saved in the manifest file: the position of the vehicle, the location of LIDAR data in S3, the location of associated pictures in S3, and so on. For large datasets, the whole process is a significant workload, and you could run it on a managed service such as Amazon SageMaker Processing, Amazon EMR or AWS Glue. Labeling 3D Point Clouds with Amazon SageMaker Ground Truth Let’s do a quick demo, based on this notebook. Starting from pre-processed sample frames, it streamlines the process of creating a 3D point cloud labeling job for each of the six task types (Object Detection, Object Tracking, Semantic Segmentation, and the associated adjustment task types). You can easily make yourself a private worker, and start labeling frames with the worker GUI and its labeling tools. A picture is worth a thousand words, and a video even more! In this first video, I annotate a couple of cars using two assistive labeling features. First, I fit the box to the ground, which helps me capture object points that are close to the ground without actually capturing the ground itself. Second, I fit the box to the object, which ensures a tight fit without any blank space. In this second video, I annotate a third car using the same technique. It’s quite harder to “see” than the previous ones, but I still manage to fit a tight box around it. Playing the next nine frames, I see that this car is actually moving. Jumping directly to the tenth frame, I adjust the bounding box to the new location of the car. Ground Truth automatically labels the eight middle frames, another assistive labeling feature called interpolation. I’ve barely scratched the surface, and there’s plenty more to learn. Now it’s your turn! Getting Started You can start labeling 3D point clouds with Amazon Sagemaker Ground Truth today in the following regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Canada (Central), Europe (Ireland), Europe (London), Europe (Frankfurt), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Seoul), Asia Pacific (Sydney), Asia Pacific (Tokyo). We’re looking forward to reading your feedback. You can send it through your usual support contacts, or in the AWS Forum for Amazon SageMaker. - Julien

New – Amazon EC2 C5a Instances Powered By 2nd Gen AMD EPYC™ Processors

Over the last 18 months, we have launched AMD-powered M5a and R5a/M5ad and R5ad, and T3a instances to provide customers additional choice for running their general purpose and memory intensive workloads. Built on the AWS Nitro System, these instances are powered by custom 1st generation AMD EPYC processors. These instances are priced 10% lower than comparable EC2 M5, R5, and T3 instances, and provide you with options to balance your instance mix based on cost and performance. Today, I am excited to announce the general availability of compute-optimized C5a instances featuring 2nd Gen AMD EPYC processors, running at frequencies up to 3.3 GHz. C5a instances are variants of Amazon EC2’s compute-optimized (C5) instance family and provide high performance processing at 10% lower cost over comparable instances. C5a instances are ideal for a broad set of compute-intensive workloads including batch processing, distributed analytics, data transformations, log analysis, and web applications. You can launch C5a instances today in eight sizes in the US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Sydney), and Asia Pacific (Singapore) Regions in On-Demand, Spot, and Reserved Instance or as part of a Savings Plan. Here are the specs: Instance Name vCPUs RAM EBS-Optimized Bandwidth Network Bandwidth c5a.large 2 4 GiB Up to 3.170 Gbps Up to 10 Gbps c5a.xlarge 4 8 GiB Up to 3.170 Gbps Up to 10 Gbps c5a.2xlarge 8 16 GiB Up to 3.170 Gbps Up to 10 Gbps c5a.4xlarge 16 32 GiB Up to 3.170 Gbps Up to 10 Gbps c5a.8xlarge 32 64 GiB 3.170 Gbps 10 Gbps c5a.12xlarge 48 96 GiB 4.750 Gbps 12 Gbps c5a.16xlarge 64 128 GiB 6.3 Gbps 20 Gbps c5a.24xlarge 96 192 GiB 9.5 Gbps 20 Gbps But wait, there’s more! Disk variants, C5ad, that come with fast, local NVMe instance storage and bare metal variants, C5an.metal and C5adn.metal, are coming soon. The C5a instances will soon be supported across a broad set of AWS services that support C5 today, including AWS Batch, Amazon EMR, Elastic Container Service (ECS), and Elastic Kubernetes Service(EKS). C5a are fully-compatible 64-bit x86 and managed by the same Nitro platform used across Amazon EC2. Again, these instances are available in similar sizes as the C5 instances, and the AMIs work on either, so go ahead and try both! To learn more, visit our AMD Instances page and please send feedback to ec2-amd-customer-feedback@amazon.com, AWS forum for EC2 or through your usual AWS Support contacts. — Channy;

Amazon FSx for Windows File Server – Storage Size and Throughput Capacity Scaling

Amazon FSx for Windows File Server provides fully managed, highly reliable file storage that is accessible over the Server Message Block (SMB) protocol. It is built on Windows Server, delivering a wide range of administrative features such as user quotas, end-user file restore, and Microsoft Active Directory integration, consistent with operating an on-premises Microsoft Windows file server. Today, we are happy to announce two new features: storage capacity scaling and throughput capacity scaling. The storage capacity scaling allows you to increase your file system size as your data set increases, and throughput capacity is bidirectional letting you adjust throughput up or down dynamically to help fine-tune performance and reduce costs. With the capability to grow storage capacity, you can adjust your storage size as your data sets grow, so you don’t need to worry about growing data sets when creating the file system. With the capability to change throughput capacity, you can dynamically adjust throughput capacity for cyclical workloads or for one-time bursts to achieve a time-sensitive goal such as data migration. When we create a file system, we specify Storage Capacity and Throughput Capacity. The storage capacity of SSD can be specified between 32 GiB and 65,536 GiB, and the capacity of HDD can be specified between 2,000 GiB and 65,536 GiB. With throughput capacity, every Amazon FSx file system has a throughput capacity that you configure when the file system is created. The throughput capacity determines the speed at which the file server hosting your file system can serve file data to clients accessing it. Higher levels of throughput capacity also come with more memory for caching data on the file server and support higher levels of IOPS. With this release, you can scale up storage capacity and can scale up / down throughput capacity on your file system with the click of a button within the AWS Management Console, or you can use the AWS Software Development Kit (SDK) or Command Line Interface (CLI) tools. The file system is available online while scaling is in progress and you’ll have full access to it for storage scaling. During scaling throughput, Amazon FSx for Windows switches out the file servers on your file system, so you’ll see an automatic failover and failback on multi-AZ file systems. So, let’s have a little trip through the new feature. We’ll look at the AWS Management Console at first. Operation by AWS Management Console Before we begin, we assume AWS Managed Microsoft AD by AWS Directory Service and Amazon FSx for Windows File Server are already set up. You can obtain a walkthrough guide here. With Actions drop down, we can select Update storage capacity and Update throughput capacity We can assign new storage capacity by Percentage or Absolute value. With throughput scaling, we can select the desired capacity from the drop down list. Then, Status is changed to In Progress, and you still have access to the file system. Scaling Storage Capacity and Throughput Capacity via CLI First, we need a CLI environment. I prefer to work on AWS Cloud9, but you can use whatever you want. We need to know the file system ID to scale it. Type in the command below: aws fsx --endpoint-url <endpoint> describe-file-systems The endpoint differs among AWS Regions, and you can get a full list here. We’ll get a return, which is long and detailed. The file system ID is at the top of the return. Let’s change Storage Capacity. The command below is the one to change it: aws fsx --endpoint-url <endpoint> update-file-system --file-system-id=<FileSystemId> --storage-capacity <new capacity> The <new capacity> should be a number up to 65536, and the new assigned capacity should be at least 10% larger than the current capacity. Once we type in the command, the new capacity is available for use within minutes. Once the new storage capacity is available on our file system, Amazon FSx begins storage optimization, which is the process of migrating the file system’s data to the new, larger disks. If needed, we can accelerate the storage optimization process at any time by temporarily increasing the file system’s throughput capacity. There is minimal performance impact while Amazon FSx performs these operations in the background, and we always have full access to our file system. If you enter the following command, you’ll see that file system update is in “IN_PROGESS” and storage optimization is in “PENDING” at the bottom part of the log return. aws fsx --endpoint-url <endpoint> describe-file-systems After the storage optimization process begins: We can also go further and run throughput scaling at the same time. Type the command below: aws fsx --endpoint-url <endpoint> update-file-system --file-system-id=<FileSystemId> --windows-configuration ThroughputCapacity=<new capacity> The “new capacity” should be <8 or 16 or 32 or 64 or 128 or 256 or 512 or 1024 or 2048> and should be larger than the current capacity. Now, we can see that throughput scaling and storage optimization are both in progress. Again, we still have full access to the file system. When we need further large capacity more than 65,536 GiB, we can use Microsoft’s Distributed File System (DFS) Namespaces to group multiple file systems under a single namespace. Available Today Storage capacity scaling and throughput capacity scaling are available today for all AWS Regions where Amazon FSx for Windows File Server is available. This support is available for new file systems starting today, and will be expanded to all file system in the coming weeks. Check our documentation for more details. – Kame;

New – SaaS Contract Upgrades and Renewals for AWS Marketplace

AWS Marketplace currently contains over 7,500 listings from 1,500 independent software vendors (ISVs). You can browse the digital catalog to find, test, buy, and deploy software that runs on AWS: Each ISV sets the pricing model and prices for their software. There are a variety of options available, including free trials, hourly or usage-based pricing, monthly, annual AMI pricing, and up-front pricing for 1-, 2-, and 3-year contracts. These options give each ISV the flexibility to define the models that work best for their customers. If their offering is delivered via a Software as a Service (SaaS) contract model, the seller can define the usage categories, dimensions, and contract length. Upgrades & Renewals AWS customers that make use of the SaaS and usage-based products that they find in AWS Marketplace generally start with a small commitment and then want to upgrade or renew them early as their workloads expand. Today we are making the process of upgrading and renewing these contracts easier than ever before. While the initial contract is still in effect, buyers can communicate with sellers to negotiate a new Private Offer that best meets their needs. The offer can include additional entitlements to use the product, pricing discounts, a payment schedule, a revised contract end-date, and changes to the end-user license agreement (EULA), all in accord with the needs of a specific buyer. Once the buyer accepts the offer, the new terms go in to effect immediately. This new, streamlined process means that sellers no longer need to track parallel (paper and digital) contracts, and also ensures that buyers receive continuous service. Let’s say I am already using a product from AWS Marketplace and negotiate an extended contract end-date with the seller. The seller creates a Private Offer for me and sends me a link that I follow in order to find & review it: I select the Upgrade offer, and I can see I have a new contract end date, the number of dimensions on my upgrade contract, and the payment schedule. I click Upgrade current contract to proceed: I confirm my intent: And I am good to go: This feature is available to all buyers & SaaS sellers, and applies to SaaS contracts and contracts with consumption pricing. — Jeff;

Single Sign-On between Okta Universal Directory and AWS

Enterprises adopting the AWS Cloud want to effectively manage identities. Having one central place to manage identities makes it easier to enforce policies, to manage access permissions, and to reduce the overhead by removing the need to duplicate users and user permissions across multiple identity silos. Having a unique identity also simplifies access for all of us, the users. We all have access to multiple systems, and we all have troubles to remember multiple distinct passwords. Being able to connect to multiple systems using one single combination of user name and password is a daily security and productivity gain. Being able to link an identity from one system with an identity managed on another trusted system is known as “Identity Federation“, which single sign-on is a subset of. Identity Federation is made possible thanks to industry standards such as Security Assertion Markup Language (SAML), OAuth, OpenID and others. Recently, we announced a new evolution of AWS Single Sign-On, allowing you to link AWS identities with Azure Active Directory identities. We did not stop there. Today, we are announcing the integration of AWS Single Sign-On with Okta Universal Directory. Let me show you the experience for System Administrators, then I will demonstrate the single sign-on experience for the users. First, let’s imagine that I am an administrator for an enterprise that already uses Okta Universal Directory to manage my workforce identities. Now I want to enable a simple and easy to use access to our AWS environments for my users, using their existing identities. As most enterprises, I manage multiple AWS Accounts. I want more than just a single sign-on solution, I want to manage access to my AWS Accounts centrally. I do not want to duplicate my Okta groups and user memberships by hand, nor maintain multiple identity systems (Okta Universal Directory and one for each AWS Account I manage). I want to enable automatic user synchronization between Okta and AWS. My users will sign in to the AWS environments using the experience they are already familiar with in Okta. Connecting Okta as an identity source for AWS Single Sign-On The first step is to add AWS Single Sign-On as an “application” Okta users can connect to. I navigate to the Okta administration console and login with my Okta administrator credentials, then I navigate to the Applications tab. Note that if I had an Okta Developer account, I’d first need to switch from Developer Console to Classic UI in the top-left navigation bar. I click the green Add Application button and I search for AWS SSO application. I click Add. I enter a name to the app (you can choose whatever name you like) and click Done. On the next screen, I configure the mutual agreement between AWS Single Sign-On and Okta. I first download the SAML Meta Data file generated by Okta by clicking the blue link Identity Provider Metadata. I keep this file, I need it later to configure the AWS side of the single sign-on. Now that I have the metadata file, I open to the AWS Management Console in a new tab. I keep the Okta tab open as the procedure is not finished there yet. I navigate to AWS Single Sign-On and click Enable AWS SSO. I click Settings in the navigation panel. I first set the Identity source by clicking the Change link and selecting External identity provider from the list of options. Secondly, I browse to and select the XML file I downloaded from Okta in the Identity provider metadata section. I click Next: Review, enter CONFIRM in the provided field, and finally click Change identity source to complete the AWS Single Sign-On side of the process. I take note of the two values AWS SSO ACS URL and AWS SSO Issuer URL as I must enter these in the Okta console. I return to the tab I left open to my Okta console, and copy the values for AWS SSO ACS URL and AWS SSO Issuer URL. I click Save to complete the configuration Configuring Automatic Provisioning Now that Okta is configured for single sign-on for my users to connect using AWS Single Sign-On I’m going to enable automatic provisioning of user accounts. As new accounts are added to Okta, and assigned to the AWS SSO application, a corresponding AWS Single Sign-On user is created automatically. As an administrator, I do not need to do any work to configure a corresponding account in AWS to map to the Okta user. From the AWS Single Sign-On Console, I navigate to Settings and then click the Enable identity synchronization link. This opens a dialog containing the values for the SCIM endpoint and an OAuth bearer access token (hidden by default). I need both of these values to use in the Okta application settings. I switch back to the tab open on the Okta console, and click on Provisioning tab under the AWS SSO Application. I select Enable API Integration. Then I copy / paste the values Base URL (I paste the value copied in AWS Single Sign-On Console SCIM endpoint) and API Token (I paste the value copied AWS Single Sign-On Console Access token). I don’t want you to be blocked like I was : be sure to remove the trailing / at the end of the URL. The URL ends with scim/v2 I click Test API Credentials to verify everything works as expected. Then I click To App to enable users creation, update, and deactivate. With provisioning enabled, my final task is to assign the users and groups that I want to synchronize from Okta to AWS Single Sign-On. I click the Assignments tab and add Okta users and groups. I click Assign, and I select the Okta users and groups I want to have access to AWS. These users are synchronized to AWS Single Sign-On, and the users now see the AWS Single Sign-On application appear in their Okta portal. To verify user synchronization is working, I switch back to the AWS Single Sign-On console and select the Users tab. The users I assigned in Okta console are present. I Configured Single Sign-On, Now What? Okta is now my single source of truth for my user identities and their assignment into groups, and periodic synchronization automatically creates corresponding identities in AWS Single Sign-On. My users sign into their AWS accounts and applications with their Okta credentials and experience, and don’t have to remember an additional user name or password. However, as things stand my users have only access to sign in. To manage permissions in terms of what they can access once signed into AWS, I must set up permissions in AWS Single Sign-On. Back to AWS SSO Console, I click AWS Accounts on the left tab bar and select the account from my AWS Organizations that I am giving access to. For enterprises having multiple accounts for multiple applications or environment, it gives you the granularity to grant access to a subset of your AWS accounts. I click Assign users to assign SSO users or groups to a set of IAM permissions. For this example, I assign just one user, the one with @example.com email address. I click Next: Permission sets and Create new permission set to create a set of IAM policies to describe the set of permissions I am granting to these Okta users. For this example, I am granting a read-only permission on all AWS services.And voila, I am ready to test this setup. SSO User Experience for the console Now that I showed you the steps System Administrators take to configure the integration, let me show you what is the user experience. As an AWS Account user, I can sign-in on Okta and get access to my AWS Management Console. I can start either from the AWS Single Sign-On user portal (the URL is on the AWS Single Sign-On settings page) or from the Okta user portal page and select the AWS SSO app. I choose to start from the AWS SSO User Portal. I am redirected to the Okta login page. I enter my Okta credentials and I land on the AWS Account and Role selection page. I click on AWS Account, select the account I want to log into, and click Management console. After a few additional redirections, I land on the AWS Console page. SSO User Experience for the CLI System administrators, DevOps engineers, Developers, and your automation scripts are not using the AWS console. They use the AWS Command Line Interface (CLI) instead. To configure SSO for the command line, I open a terminal and type aws configure sso. I enter the AWS SSO User Portal URL and the Region. $aws configure sso SSO start URL [None]: https://d-0123456789.awsapps.com/start SSO Region [None]: eu-west-1 Attempting to automatically open the SSO authorization page in your default browser. If the browser does not open or you wish to use a different device to authorize this request, open the following URL: https://device.sso.eu-west-1.amazonaws.com/ Then enter the code: AAAA-BBBB At this stage, my default browser pops up and I enter my Okta credentials on the Okta login page. I confirm I want to enable SSO for the CLI. and I close the browser when I receive this message: The CLI automatically resumes the configuration, I enter the default Region, the default output format and the name of the CLI profile I want to use. The only AWS account available to you is: 012345678901 Using the account ID 012345678901 The only role available to you is: ViewOnlyAccess Using the role name "ViewOnlyAccess" CLI default client Region [eu-west-1]: CLI default output format [None]: CLI profile name [okta]: To use this profile, specify the profile name using --profile, as shown: aws s3 ls --profile okta I am now ready to use the CLI with SSO. In my terminal, I type: aws --profile okta s3 ls 2020-05-04 23:14:49 do-not-delete-gatedgarden-audit-012345678901 2015-09-24 16:46:30 elasticbeanstalk-eu-west-1-012345678901 2015-06-11 08:23:17 elasticbeanstalk-us-west-2-012345678901 If the machine you want to configure CLI SSO has no graphical user interface, you can configure SSO in headless mode, using the URL and the code provided by the CLI (https://device.sso.eu-west-1.amazonaws.com/ and AAAA-BBBB in the example above) In this post, I showed how you can take advantage of the new AWS Single Sign-On capabilities to link Okta identities to AWS accounts for user single sign-on. I also make use of the automatic provisioning support to reduce complexity when managing and using identities. Administrators can now use a single source of truth for managing their users, and users no longer need to manage an additional identity and password to sign into their AWS accounts and applications. AWS Single Sign-On with Okta is free to use, and is available in all Regions where AWS Single Sign-On is available. The full list is here. To see all this in motion, you can check out the following demo video for more details on getting started. -- seb

Pages

Recommended Content