'How to Accelerate Auto Data Tagging for Autonomous Driving in Hybrid Cloud' by Alluxio spark datatagging
In this blog, guest writer Feifei Cai , Hao Zhu , shares the practice of using Alluxio and Spark to accelerate the auto data tagging system in WeRide, an autonomous driving technology company. The original content was published on Alluxio's Blog .is a global leading company that develops L4 autonomous driving technologies.
We have both hardware and software in the loop. The biggest challenge is the size of the data. We generate many terabytes of data every day and have accumulated millions of kilometers of mileage, so the entire size is in petabytes. This requires vast amounts of storage and significant computing power.Tags are sometimes confused with labels. In autonomous driving, tags usually mean scenarios, while labels can be obstacles or boxes.
In a data-driven autonomous driving workflow, data tagging plays an essential role in data analytics. With the tagging results, we can do data selection to support the downstream pipelines, including model training, benchmark, and long-tail evaluation.An end-to-end tagging pipeline is to collect and upload the ROS bag files from the test vehicles and then convert them into Parquet format. After that, we can run Spark jobs to perform data analytics algorithms.
In both public cloud and on-prem data centers, we use Kubernetes to manage the computing resources of the CPU and the GPU clusters and to schedule the workflow and tasks. In the storage layer, we use MinIO to build S3 compatible object storage service.
United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Decentralized Computing & Storage vs. Legacy Cloud Solutions | HackerNoon'Decentralized Computing & Storage vs. Legacy Cloud Solutions' by aleph_im decentralization cloudcomputing
Read more »
'Humble and hidden' auto suppliers driving the EV revolutionThe shift to electric cars may pose an existential threat to suppliers of combustion engines but for auto parts firms such as TE Connectivity the challenge is keeping up with demand.
Read more »
What is Play to Earn and Why You Should Join the Movement | HackerNoonPlay to Earn allow gamers to build a career out of gaming and truly own all the digital assets inside their favorite virtual worlds.
Read more »
How to Market on Quora Effectively in 2022 | HackerNoonQuora is packed with high-intent users, which are a goldmine for any marketer. Find out how it works, and how you can use it as a marketing channel.
Read more »
Shopify Store Backup Strategy: 3 Ways to Keep Your Business Protected | HackerNoonThis article will discuss some ways you can backup your Shopify website and protect your business from potential disasters.
Read more »