Backend Batch Processing with Nomad смотреть онлайн

Jason Hancock, Operations Engineer at Graymeta

At Graymeta, we index large filesystems by extracting metadata and leveraging machine learning APIs to process files. Wanting to benefit from the security of running our indexing process in a container, reduce our code’s complexity that was scheduling jobs to run on specific nodes, consolidate infrastructure by treating all back-end batch processing machines as a pool of resources to scale up and down as a single unit, and several other reasons, we began looking for an easy to operate container orchestration layer. For us, it has to be dead simple to operate as we have to be able to run it in the cloud and behind corporate firewalls in on-premises deployments. We chose Nomad as it was the best fit for our requirements.

This talk will dive into the how of moving our application from the traditional processing jobs out of a queue on multiple VMs to processing the same jobs out of the same queue as before, but scheduling them as container jobs in Nomad. It will also touch on our local development environments that each developer runs on their laptop where we run Nomad inside a Docker container and use the raw_exec driver to mimic what we’re doing in production. As a final point, I will discuss the increased portability and flexibility that moving to Nomad has afforded us, specifically in terms of being able to abstract the container scheduling and use a variety of different schedulers in different environments (for example, ECS in EC2). --

If you liked this video and want to see more from HashiCorp, subscribe to our channel: https://www.youtube.com/c/HashiCorp?sub_confirmation=1

To learn more, visit our hands-on interactive lab environment, HashiCorp Learn: https://learn.hashicorp.com/

HashiCorp is the leader in multi-cloud infrastructure automation software. The HashiCorp software suite enables organizations to adopt consistent workflows to provision, secure, connect, and run any infrastructure for any application. HashiCorp open source tools Vagrant, Packer, Terraform, Vault, Consul, Nomad, Boundary, and Waypoint are downloaded tens of millions of times each year and are broadly adopted by the Global 2000. Enterprise versions of these products enhance the open source tools with features that promote collaboration, operations, governance, and multi-data center functionality.

For more information, visit: www.hashicorp.com or follow us on social media:
Twitter: @hashicorp
LinkedIn: https://www.linkedin.com/company/hashicorp
Facebook: https://www.facebook.com/HashiCorp