ishan.sh

pi-computer-use Port

2026-05-03T00:00:00.000Z

used pi and gpt 5.5 to extend the pi-computer-use extension for pi to also work on ubuntu. I spent about 10 minutes working on this.

GitHub repository ishandotsh/pi-computer-use View the repository on GitHub.

Just an experiment (probably) never working on this again

Download video

Silly demo

Sorry, a better demo:

Download video

Spotify demo

Building with Local Qwen3.6

2026-05-03T00:00:00.000Z

I've spent the last couple of weeks playing around with running local models on my RTX3090 24gb. I tried doing a similar thing about an year ago but local models were nowhere near as useful as they are now and I was left pretty disappointed.

Thanks to a bunch of people its been really easy getting started:

Sudo:
Club3090: https://github.com/noonghunna/club-3090

But people on twitter post metrics that are engineered to grab your attention, and are not representative of how these models perform on tasks that I'd like to do. Example:

Its cool, but its useless. Also, hermes is a pretty good full-featured harness, but my overwhelmed self likes the simpler Pi.

Here's a weekend experiment I ran.

Setup

How I ran Qwen3.6-27b:

./build/bin/llama-server \
  -hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M \
  -ngl 99 \
  -c 131072 \
  -np 1 \
  -fa on \
  -ctk q8_0 \
  -ctv turbo3 \
  --metrics \
  --jinja --chat-template-kwargs '{"enable_thinking": false}'

Context length of 128K tokens, flash attention, asymmetric KV cache with key quantization of q8_0 and value quantization set to Turbo3 (thanks to Tom Turney). Also I turned thinking off. I've noticed that Qwen models seem to overthink and get stuck in loops. It also fills the context window that pushes these models to the "dumb zone" quicker.

I used Matt Pocock's /grill-me skill to get to a shared understanding of the plan for this website. I ran it through Qwen and here's the entire conversation I had with it:

I gave the exact same plan to gpt5.5 and qwen3.6 running via the pi-coding-agent harness. The full conversations are here:

Results

GPT5.5 obviously nailed it first try. And it completed the task the fastest.

Qwen3.6's implementation was pretty good. It made a lot of mistakes, took way longer to get there, but it got there! It wasn't fun or delightful like I've come to expect from frontier labs, but my PC isn't exactly a frontier lab. It looked like this (notice the spelling mistake "Ischan" in the subheading):

And if you needed any proof that turning thinking off was the right move, here's what Qwen3.6+Thinking built with the exact same prompt and plan:

I (gpt-5.5) built a small tool to track llamacpp's performance. The performance was okay in the beginning but as context increased it slowed way down:

Run length: 4142s wall (~69 min). 3942s active, 200s idle.
Output: 46,054 tokens generated, 277,568 prompt tokens processed.
Decode rate degraded over the run:
- First sample: 28.08 tok/s
- Final sample: 16.19 tok/s (-42%)
- First-half average: 23.44 tok/s
- Second-half average: 17.13 tok/s
- Min: 16.19, Max: 28.09 (across 403 ten-second samples)

As the KV cache fills, decode gets cheaper-per-step but slower-per-token, and the re-prompting (qwen kept re-reading files) of long files makes prompt-eval the dominant cost. By the back half of the run, throughput is below 60% of the cold-start rate. And Qwen made a lot of smaller mistakes. Claude Opus 4.6 (xhigh)'s analysis of the performance and code differences

Conclusion

Using local models reminds me of when I was younger playing GTA-V on my old PC, slightly changing settings to maximize how good the game looked vs having a playable FPS. Using frontier models is like when I got a good enough PC to not care about all that and got on with playing my game.

Isaac Sim Accelerated: Part 3

2024-07-03T00:00:00.000Z

Git Repository: https://github.com/ishandotsh/isaac-sim-starter

Recap

Once all sensors are attached to the robot from Part 2, we can access the sensor data through ROS using action graphs.

Connecting to ROS (ActionGraph)

With the environment, robot, and sensors ready we can start publishing topics. The way to do that in Isaac is to use action graphs - similar to Unity, but for robots.

Go through the ROS tutorials on the Isaac Sim documentation or you’ll feel lost in this section: https://docs.omniverse.nvidia.com/isaacsim/latest/ros_tutorials/index.html

BTW, this is the cheatsheet part of the article. Writing out these actiongraphs is kind of tedious.

Cameras

Let’s build an actiongraph that publishes the camera’s image to /camera/color and camera info to /camera/camera_info. Here’s how to do it using the GUI (annoying): https://docs.omniverse.nvidia.com/isaacsim/latest/ros_tutorials/tutorial_ros_camera.html

In python (also annoying, but less mouse clicking):

# Enable ROS1 Bridge extension (for ROS1 Nodes)
from omni.isaac.core.utils.extensions import enable_extension
enable_extension("omni.isaac.ros_bridge")

import omni.graph.core as og
import usdrt.Sdf
controller = og.Controller
keys = controller.Keys

camera_graph_config = {"graph_path": "/CameraActionGraph", "evaluator_name": "execution"}
# https://docs.omniverse.nvidia.com/kit/docs/omni.graph/latest/omni.graph.core/omni.graph.core.Controller.html#omni.graph.core.Controller.edit

camera_graph, _, _, _ = controller.edit(
     camera_graph_config,
     {
          keys.CREATE_NODES: [
               ("OnPlaybackTick", "omni.graph.action.OnPlaybackTick"),
               ("CreateRenderProduct",  "omni.isaac.core_nodes.IsaacCreateRenderProduct"),
               ("CameraHelperRgb",   "omni.isaac.ros_bridge.ROS1CameraHelper"),
               ("CameraHelperInfo",  "omni.isaac.ros_bridge.ROS1CameraHelper"),
          ],
          keys.CONNECT: [
               ("OnPlaybackTick.outputs:tick", "CreateRenderProduct.inputs:execIn"),
               ("CreateRenderProduct.outputs:execOut", "CameraHelperRgb.inputs:execIn"),
               ("CreateRenderProduct.outputs:execOut", "CameraHelperInfo.inputs:execIn"),
               ("CreateRenderProduct.outputs:renderProductPath", "CameraHelperRgb.inputs:renderProductPath"),
               ("CreateRenderProduct.outputs:renderProductPath", "CameraHelperInfo.inputs:renderProductPath"),
          ],
          keys.SET_VALUES: [
               ("CreateRenderProduct.inputs:cameraPrim", [usdrt.Sdf.Path(camera_path)]),
               ("CameraHelperRgb.inputs:frameId", "camera"),
               ("CameraHelperRgb.inputs:topicName", "camera/color"),
               ("CameraHelperRgb.inputs:type", "rgb"),
               ("CameraHelperInfo.inputs:frameId", "camera"),
               ("CameraHelperInfo.inputs:topicName", "camera/camera_info"),
               ("CameraHelperInfo.inputs:type", "camera_info"),
          ]     
     }
)

$ rostopic list
/camera/camera_info
/camera/color
/rosout
/rosout_agg

In RVIZ:

What I like to do is build the simplest graph then add nodes using helper functions. All helper functions are available at utils/utils.py in the repo.

Here’s the simplest graph with a OnPlaybackTick node, and a publish ROS Clock node:

graph_config = {"graph_path": "/ActionGraph", "evaluator_name": "execution"}
graph, _, _, _ = controller.edit(
     graph_config,
     {
          keys.CREATE_NODES: [
               ("OnPlaybackTick", "omni.graph.action.OnPlaybackTick"),
               ("ReadSimTime", "omni.isaac.core_nodes.IsaacReadSimulationTime"),
               ("PublishClock", "omni.isaac.ros_bridge.ROS1PublishClock"),
          ],
          og.Controller.Keys.CONNECT: [
               ("OnPlaybackTick.outputs:tick", "PublishClock.inputs:execIn"),
               ("ReadSimTime.outputs:simulationTime", "PublishClock.inputs:timeStamp"),
          ],
          og.Controller.Keys.SET_VALUES: [
               ("PublishClock.inputs:topicName", "/clock"),
          ]
     }
)

Then use helper functions to attach any number of cameras/lidars. Note that these don't "add" the sensor to the robot, they add the action graph nodes to the action graph.

def add_camera(
        graph: og.Graph,
        camera_path, # full path to camera prim
        camera_name, # node identifier (not related to ros)
        colorFrameId,
        colorTopicName,
        infoFrameId,
        infoTopicName,
        depthFrameId = None,
        depthTopicName = None):

    isDepthCam = False
    if(type(depthFrameId) != type(depthTopicName)):
        print("You must provide both depthFrameId and depthTopicName to create a depth camera")
        print("Skipping creating depth camera")
        isDepthCam = False
    if(type(depthFrameId) == str):
        isDepthCam = True

    graph_path = graph.get_path_to_graph()
    tick_node_path = graph_path + '/OnPlaybackTick'
    create_render_path = graph_path + '/CreateRenderProduct' + camera_name
    rgb_path = graph_path + '/CameraHelperRgb' + camera_name
    info_path = graph_path + '/CameraHelperInfo' + camera_name
    
    controller = og.Controller
    controller.create_node(create_render_path, "omni.isaac.core_nodes.IsaacCreateRenderProduct")
    controller.create_node(rgb_path,    "omni.isaac.ros_bridge.ROS1CameraHelper")
    controller.create_node(info_path,   "omni.isaac.ros_bridge.ROS1CameraHelper")

    controller.connect(tick_node_path+'.outputs:tick',create_render_path+'.inputs:execIn')
    controller.connect(create_render_path+'.outputs:execOut',rgb_path+'.inputs:execIn')
    controller.connect(create_render_path+'.outputs:execOut',info_path+'.inputs:execIn')
    controller.connect(create_render_path+'.outputs:renderProductPath', rgb_path+'.inputs:renderProductPath')
    controller.connect(create_render_path+'.outputs:renderProductPath', info_path+'.inputs:renderProductPath')

    controller.attribute(create_render_path+'.inputs:cameraPrim').set([usdrt.Sdf.Path(camera_path)])
    controller.attribute(rgb_path+'.inputs:frameId').set(colorFrameId)
    controller.attribute(info_path+'.inputs:frameId').set(infoFrameId)
    controller.attribute(rgb_path+'.inputs:topicName').set(colorTopicName)
    controller.attribute(info_path+'.inputs:topicName').set(infoTopicName)
    controller.attribute(rgb_path+'.inputs:type').set("rgb")
    controller.attribute(info_path+'.inputs:type').set("camera_info")

    if(isDepthCam):
        depth_path = graph_path + '/CameraHelperDepth' + camera_name
        controller.create_node(depth_path,   "omni.isaac.ros_bridge.ROS1CameraHelper")
        controller.connect(create_render_path+'.outputs:execOut',depth_path+'.inputs:execIn')
        controller.connect(create_render_path+'.outputs:renderProductPath', depth_path+'.inputs:renderProductPath')
        controller.attribute(depth_path+'.inputs:frameId').set(depthFrameId)
        controller.attribute(depth_path+'.inputs:topicName').set(depthTopicName)
        controller.attribute(depth_path+'.inputs:type').set("depth")

Lets add the camera through this helper function:

add_camera(graph, camera_path=camera_path, camera_name="my_camera",
                    colorFrameId="camera", colorTopicName="camera/color",
          infoFrameId="camera", infoTopicName="camera/camera_info")

Lidar

def add_lidar(
        graph: og.Graph,
        lidar_path,
        lidar_name,
        lidarFrameId,
        lidarTopicName,
        lidarType = "laser_scan" # or point_cloud
    ):
    graph_path = graph.get_path_to_graph()
    tick_node_path = graph_path + '/OnPlaybackTick'
    create_render_path = graph_path + '/CreateRenderProduct' + lidar_name
    lidar_helper_path = graph_path + '/LidarHelper' + lidar_name

    controller = og.Controller
    controller.create_node(create_render_path, "omni.isaac.core_nodes.IsaacCreateRenderProduct")
    controller.create_node(lidar_helper_path,    "omni.isaac.ros_bridge.ROS1RtxLidarHelper")

    controller.connect(tick_node_path+'.outputs:tick',create_render_path+'.inputs:execIn')
    controller.connect(create_render_path+'.outputs:execOut',lidar_helper_path+'.inputs:execIn')
    controller.connect(create_render_path+'.outputs:renderProductPath', lidar_helper_path+'.inputs:renderProductPath')

    controller.attribute(create_render_path+'.inputs:cameraPrim').set([usdrt.Sdf.Path(lidar_path)])
    controller.attribute(lidar_helper_path+'.inputs:frameId').set(lidarFrameId)
    controller.attribute(lidar_helper_path+'.inputs:topicName').set(lidarTopicName)
    controller.attribute(lidar_helper_path+'.inputs:type').set(lidarType)

add_lidar(graph, lidar_parent+"/"+lidar_path, "my_lidar", 
          "laser", "scan", "laser_scan")
# Notice the lidar path needs to be the full path: 
# /turtlebot3_waffle_pi/base_scan/lidar
# We defined those earlier when we added the lidar

I spawned in the Simple Room environment and this is what I see in RVIZ:

In the spirit of completeness, here’s how to publish PhysX Lidar output via actiongraph in python. This is not a helper function, if you want to use this add it to the simplest action graph we created at the start.

# This won't work if the original ActionGraph is also uncommented
og.Controller.edit(
     {"graph_path": "/ActionGraph", "evaluator_name": "execution"},
     {
          og.Controller.Keys.CREATE_NODES: [
               ("ReadSimTime", "omni.isaac.core_nodes.IsaacReadSimulationTime"),
               ("OnPlaybackTick", "omni.graph.action.OnPlaybackTick"),
               ("PublishClock", "omni.isaac.ros_bridge.ROS1PublishClock"),

               ("ReadLidar", "omni.isaac.range_sensor.IsaacReadLidarBeams"),
               ("PublishLidar", "omni.isaac.ros_bridge.ROS1PublishLaserScan"),
          ],
          og.Controller.Keys.CONNECT: [
               ("OnPlaybackTick.outputs:tick", "PublishClock.inputs:execIn"),
               ("ReadSimTime.outputs:simulationTime", "PublishClock.inputs:timeStamp"),
               ("OnPlaybackTick.outputs:tick", "ReadLidar.inputs:execIn"),
               ("ReadSimTime.outputs:simulationTime", "PublishLidar.inputs:timeStamp"),
               ("ReadLidar.outputs:execOut",               "PublishLidar.inputs:execIn"),
               ("ReadLidar.outputs:azimuthRange",          "PublishLidar.inputs:azimuthRange"),
               ("ReadLidar.outputs:depthRange",            "PublishLidar.inputs:depthRange"),
               ("ReadLidar.outputs:horizontalFov",         "PublishLidar.inputs:horizontalFov"),
               ("ReadLidar.outputs:horizontalResolution",  "PublishLidar.inputs:horizontalResolution"),
               ("ReadLidar.outputs:intensitiesData",       "PublishLidar.inputs:intensitiesData"),
               ("ReadLidar.outputs:linearDepthData",       "PublishLidar.inputs:linearDepthData"),
               ("ReadLidar.outputs:numCols",               "PublishLidar.inputs:numCols"),
               ("ReadLidar.outputs:numRows",               "PublishLidar.inputs:numRows"),
               ("ReadLidar.outputs:rotationRate",          "PublishLidar.inputs:rotationRate"),
          ],
          og.Controller.Keys.SET_VALUES: [
               ("PublishClock.inputs:topicName", "/clock"),
               ("ReadLidarB.inputs:lidarPrim", "/turtlebot3_waffle_pi/base_scan/lidar"),
               ("PublishLidarB.inputs:frameId", "laser"),
               ("PublishLidarB.inputs:topicName", "scan"),
          ]
     }
)

IMU

def add_imu(graph, imu_path, imuFrameId, imuTopicName):
     graph_path = graph.get_path_to_graph()
     tick_node_path = graph_path + '/OnPlaybackTick'
     ros_clock_node_path = graph_path + '/ReadSimTime'
     read_imu_path = graph_path + '/ReadIMU'
     pub_imu_path = graph_path + '/PublishIMU'

     controller = og.Controller
     controller.create_node(read_imu_path, "omni.isaac.sensor.IsaacReadIMU")
     controller.create_node(pub_imu_path, "omni.isaac.ros_bridge.ROS1PublishImu")

     controller.connect(tick_node_path+'.outputs:tick', read_imu_path+'.inputs:execIn')
     controller.connect(ros_clock_node_path+'.outputs:simulationTime', pub_imu_path+'.inputs:timeStamp')
     controller.connect(read_imu_path+'.outputs:execOut', pub_imu_path+'.inputs:execIn')
     controller.connect(read_imu_path+'.outputs:linAcc', pub_imu_path+'.inputs:linearAcceleration')
     controller.connect(read_imu_path+'.outputs:angVel', pub_imu_path+'.inputs:angularVelocity')
     controller.connect(read_imu_path+'.outputs:orientation', pub_imu_path+'.inputs:orientation')

     controller.attribute(read_imu_path+'.inputs:imuPrim').set([usdrt.Sdf.Path(imu_path)])
     controller.attribute(pub_imu_path+'.inputs:frameId').set(imuFrameId)
     controller.attribute(pub_imu_path+'.inputs:topicName').set(imuTopicName)

add_imu(graph, imu_parent+'/'+imu_path, "IMU", "imu")

Teleop

This is where the concept of Articulations comes in. You have to identify the articulation root for the robot, in this case its “base_footprint”. In custom robots not imported via URDF, you’ll have to add the ArticulationAPI yourself. The joint_names parameter is the same as the ones we applied the DriveAPI to, so we can just pass that same list in. More info: https://docs.omniverse.nvidia.com/isaacsim/latest/ros_tutorials/tutorial_ros_drive_turtlebot.html

from typing import List
def add_diff_teleop(graph, robot_path, joint_names:List[str], max_linear_speed, wheel_distance, wheel_radius, topic_name):
     graph_path = graph.get_path_to_graph()
     tick_node_path = graph_path + '/OnPlaybackTick'
     sub_twist_path = graph_path + '/SubscribeTwistDiff'
     scale_path = graph_path + '/ScaleWheel'
     break_ang_path = graph_path + '/Break3_VectorAngVel'
     break_lin_path = graph_path + '/Break3_VectorLinVel'
     diff_contr_path = graph_path + '/DiffController'
     art_contr_path = graph_path + '/ArticulationController'

         controller = og.Controller
     controller.create_node(sub_twist_path, "omni.isaac.ros_bridge.ROS1SubscribeTwist")
     controller.create_node(scale_path, "omni.isaac.core_nodes.OgnIsaacScaleToFromStageUnit")
     controller.create_node(break_ang_path, "omni.graph.nodes.BreakVector3")
     controller.create_node(break_lin_path, "omni.graph.nodes.BreakVector3")
     controller.create_node(diff_contr_path, "omni.isaac.wheeled_robots.DifferentialController")
     controller.create_node(art_contr_path, "omni.isaac.core_nodes.IsaacArticulationController")

     controller.connect(tick_node_path+'.outputs:tick',sub_twist_path+'.inputs:execIn')
     controller.connect(tick_node_path+'.outputs:tick',art_contr_path+'.inputs:execIn')
     controller.connect(sub_twist_path+'.outputs:execOut',diff_contr_path+'.inputs:execIn')
     controller.connect(sub_twist_path+'.outputs:angularVelocity',break_ang_path+'.inputs:tuple')
     controller.connect(sub_twist_path+'.outputs:linearVelocity',scale_path+'.inputs:value')
     controller.connect(scale_path+'.outputs:result',break_lin_path+'.inputs:tuple')
     controller.connect(break_ang_path+'.outputs:z',diff_contr_path+'.inputs:angularVelocity')
     controller.connect(break_lin_path+'.outputs:x',diff_contr_path+'.inputs:linearVelocity')
     controller.connect(diff_contr_path+'.outputs:velocityCommand',art_contr_path+'.inputs:velocityCommand')

     controller.attribute(sub_twist_path+'.inputs:topicName').set(topic_name)
     controller.attribute(diff_contr_path+'.inputs:maxLinearSpeed').set(max_linear_speed)
     controller.attribute(diff_contr_path+'.inputs:wheelDistance').set(wheel_distance)
     controller.attribute(diff_contr_path+'.inputs:wheelRadius').set(wheel_radius)
     controller.attribute(art_contr_path+'.inputs:usePath').set(True)
     controller.attribute(art_contr_path+'.inputs:robotPath').set(robot_path)
     controller.attribute(art_contr_path+'.inputs:jointNames').set(joint_names)

add_diff_teleop(graph, str(robot_prim.GetPrimPath())+'/base_footprint', velocity_driven_joints, 0.26, 0.3, 0.066, "cmd_vel")

Odom

Publish odom→base_footprint transform on /tf and odometry on /odom:

def add_odometry(graph, ogn_node_name, chassis_prim_path, chassis_frame_id, odom_frame_id, odom_topic_name):
     graph_path = graph.get_path_to_graph()
     tick_node_path = graph_path + '/OnPlaybackTick'
     ros_clock_node_path = graph_path + '/ReadSimTime'
     compute_odom_path = graph_path + '/' + ogn_node_name
     pub_odom_path = graph_path + '/Pub' + ogn_node_name
     pub_raw_tf_path = graph_path + '/' + 'PublishRawTFTree'

     controller = og.Controller
     controller.create_node(compute_odom_path, "omni.isaac.core_nodes.IsaacComputeOdometry") 
     controller.create_node(pub_odom_path, "omni.isaac.ros_bridge.ROS1PublishOdometry")
     controller.create_node(pub_raw_tf_path, "omni.isaac.ros_bridge.ROS1PublishRawTransformTree")

     controller.connect(tick_node_path+'.outputs:tick',compute_odom_path+'.inputs:execIn')
     controller.connect(tick_node_path+'.outputs:tick',pub_odom_path+'.inputs:execIn')
     controller.connect(ros_clock_node_path+'.outputs:simulationTime',pub_odom_path+'.inputs:timeStamp')
     controller.connect(compute_odom_path+'.outputs:execOut',pub_odom_path+'.inputs:execIn')
     controller.connect(compute_odom_path+'.outputs:angularVelocity',pub_odom_path+'.inputs:angularVelocity')
     controller.connect(compute_odom_path+'.outputs:linearVelocity',pub_odom_path+'.inputs:linearVelocity')
     controller.connect(compute_odom_path+'.outputs:orientation',pub_odom_path+'.inputs:orientation')
     controller.connect(compute_odom_path+'.outputs:position',pub_odom_path+'.inputs:position')
     controller.connect(compute_odom_path+'.outputs:execOut',pub_raw_tf_path+'.inputs:execIn')
     controller.connect(compute_odom_path+'.outputs:orientation',pub_raw_tf_path+'.inputs:rotation')
     controller.connect(compute_odom_path+'.outputs:position',pub_raw_tf_path+'.inputs:translation')
     controller.connect(ros_clock_node_path+'.outputs:simulationTime',pub_raw_tf_path+'.inputs:timeStamp')

     controller.attribute(compute_odom_path+'.inputs:chassisPrim').set(chassis_prim_path)
     controller.attribute(pub_odom_path+'.inputs:chassisFrameId').set(chassis_frame_id)
     controller.attribute(pub_odom_path+'.inputs:odomFrameId').set(odom_frame_id)
     controller.attribute(pub_odom_path+'.inputs:topicName').set(odom_topic_name)
     controller.attribute(pub_raw_tf_path+'.inputs:childFrameId').set(chassis_frame_id)
     controller.attribute(pub_raw_tf_path+'.inputs:parentFrameId').set(odom_frame_id)
     controller.attribute(pub_raw_tf_path+'.inputs:topicName').set("tf")

add_odometry(graph, "ComputeOdometry", str(robot_prim.GetPrimPath())+'/base_footprint',
     "base_footprint", "odom", "odom")

TF (tf and tf_static)

from typing import Optional
def add_tf(graph, ogn_node_name, target_prims: List[str], topic_name = "tf", parent_prim: Optional[str] = None, namespace: Optional[str] = None):
     graph_path = graph.get_path_to_graph()
     tick_node_path = graph_path + '/OnPlaybackTick'
     ros_clock_node_path = graph_path + '/ReadSimTime'
     publish_tf_path = graph_path + '/' + ogn_node_name

     controller = og.Controller
     controller.create_node(publish_tf_path, "omni.isaac.ros_bridge.ROS1PublishTransformTree")

     controller.connect(tick_node_path+'.outputs:tick',publish_tf_path+'.inputs:execIn')
     controller.connect(ros_clock_node_path+'.outputs:simulationTime', publish_tf_path+'.inputs:timeStamp')

     controller.attribute(publish_tf_path+'.inputs:targetPrims').set(target_prims)
     if(namespace is not None):
          controller.attribute(publish_tf_path+'.inputs:nodeNamespace').set(namespace)
     if(parent_prim is not None):
          controller.attribute(publish_tf_path+'.inputs:parentPrim').set(parent_prim)

Isaac Sim will automatically parse links to publish the TF for the entire robot given the articulation root of the robot as the target_prim:

add_tf(graph, "PubTF", target_prims=[str(robot_prim.GetPrimPath())+'/base_link'])

This will however, publish with “world” as the root node. To prevent that, build the TF tree manually. A lot of ROS packages require publishing of static TFs on /tf_static. This is possible to do through this helper too:

# manual tf tree
# base_footprint -> base_link on /tf_static
add_tf(graph, "PubTFBaseStatic", [str(robot_prim.GetPrimPath())+'/base_link'], "tf_static", str(robot_prim.GetPrimPath())+'/base_footprint')

# defines base_link -> (all other joints) on /tf_static
static_links = [
     "camera_link", "caster_back_left_link",
     "caster_back_right_link", "imu_link", "base_scan",
]
path_prefix = str(robot_prim.GetPrimPath()) + '/'
static_links_paths = [path_prefix + link for link in static_links]

add_tf(graph, "PubTFStatic", static_links_paths, "tf_static", str(robot_prim.GetPrimPath())+'/base_link')

# base_link -> wheel links on /tf
wheel_links = ["wheel_left_link", "wheel_right_link"]
wheel_links_paths = [path_prefix + link for link in wheel_links]

add_tf(graph, "PubTFWheels", wheel_links_paths, "tf", str(robot_prim.GetPrimPath())+'/base_link')

The result is essentially the same, you can add the camera’s links back through the helper too.

Wrap up

Run the script

roscore

# in another terminal 
./python.sh ./isaac-sim-starter/isaac.py

# check ros stuff
rosnode list
rostopic list

# check RVIZ
rviz

With that, you have a working robot in Isaac Sim with the foundational work done. I believe this is a good jumping off point to build your simulation. I'll list out some features that I wanted to add to this series but haven't yet:

Custom action graph nodes
Script nodes
Cloner
Dynamic Environments

and some features that I'd like to see added to Isaac Sim:

An API for OnShape Importer
Easier method to add retro-reflective material to objects for lidars
A fix for RTX Lidar max_angle not working

If you've worked on any of the listed items and could point me to your work contact me

Isaac Sim Accelerated: Part 2

2024-07-02T00:00:00.000Z

Git Repository: https://github.com/ishandotsh/isaac-sim-starter

Recap

Make sure you've set up the base script from Part 1 and ran it.

Getting your robot in the correct format

Depending on your situation, one of these will be the way to get your robot description in USD:

Import existing URDF
Rig a model inside Isaac Sim (Rigging Robots section of the documentation)
Import a CAD model from OnShape

Import URDF

💡 IMPORTANT: Any mesh paths in the URDF need to be replaced with full paths (or you can source the workspace containing the description before launching Isaac)

Here’s how to import a URDF through python:

from omni.importer.urdf import _urdf

status, robot_import_config = omni.kit.commands.execute("URDFCreateImportConfig")
robot_import_config.merge_fixed_joints = False
robot_import_config.convex_decomp = True # Collision
robot_import_config.import_inertia_tensor = True
robot_import_config.fix_base = False
robot_import_config.distance_scale = 1

# 
robot_import_config.default_drive_type = _urdf.UrdfJointTargetType.JOINT_DRIVE_NONE

robot_urdf_path = "/PATH/isaac-sim-starter/assets/turtlebot3_waffle_pi/urdf/turtlebot3_waffle_pi.urdf" # change this
status, robot_prim_path = omni.kit.commands.execute(
     "URDFParseAndImportFile",
     urdf_path=robot_urdf_path,
     import_config=robot_import_config,
     get_articulation_root=True,
)

To import a urdf through the GUI, follow NVIDIA’s tutorial: https://docs.omniverse.nvidia.com/isaacsim/latest/ros_tutorials/tutorial_ros_turtlebot.html

💡 URDF Importer Docs:

https://docs.omniverse.nvidia.com/kit/docs/omniverse-urdf-importer/latest/source/extensions/omni.importer.urdf/docs/index.html

The next step is to set the joint properties for the castor wheels and the drive wheels. The most important ones are joint friction, max velocity, damping, and stiffness. In addition, the drive wheel joints need to have the DriveAPI attached to them.

This is the full code, breakdown is below it:

from typing import List

velocity_driven_joints = ['wheel_left_joint', 'wheel_right_joint']
position_driven_joints = []
undriven_joints = []

def get_robot_description(robot_prim: Usd.Prim) -> List[Usd.Prim]:
     robot_description = []

     def gather_paths(prim: Usd.Prim):
          for child_prim in prim.GetAllChildren():
               if child_prim.GetTypeName() not in ["Mesh", "Shader", "Material", "Scope"]:
               # print(child_prim.GetPath().pathString, " -> ", child_prim.GetTypeName())
               robot_description.append(child_prim)
               gather_paths(child_prim)

     gather_paths(robot_prim)
     return robot_description

robot_prim = stage.GetPrimAtPath(robot_prim_path).GetParent()
robot_tree = get_robot_description(robot_prim)

# Set velocity, position, or undriven joints
for prim in robot_tree:
     prim_type = prim.GetTypeName()
     prim_path = prim.GetPath()
     if prim_type == "PhysicsRevoluteJoint":
          # print("At ", prim.GetPath().pathString, end=" ")
          prim_name = prim_path.pathString.split("/")[-1].strip()
     if prim_name in velocity_driven_joints:
          # print("-> Setting Velocity")
          drive = UsdPhysics.DriveAPI.Get(prim, "angular")
          drive.GetStiffnessAttr().Set(0)
          drive.GetDampingAttr().Set(1e10)
     elif prim_name in position_driven_joints:
          # print("-> Setting Position")
          drive = UsdPhysics.DriveAPI.Get(prim, "angular")
          drive.GetStiffnessAttr().Set(1e10)
          drive.GetDampingAttr().Set(0)
     else:
          # print("-> Setting Undriven")
          # print("Prop Names: ")
          # print(prim.GetPropertyNames())
          prim.RemoveAPI(UsdPhysics.DriveAPI, "angular")
          prim.GetProperty("physxJoint:jointFriction").Set(0.001)
          prim.GetProperty("physxJoint:maxJointVelocity").Set(4.0)

Let me break it down for you

The get_robot_description method traverses the prim paths of the robot and returns a flat list of all prims. This is not strictly necessary, and a similar method is probably already implemented in the OpenUSD module but this was easier than reading the documentation for me. If you find it, let me know - I'll update this, and credit you.

def get_robot_description(robot_prim: Usd.Prim) -> List[Usd.Prim]:
     robot_description = []

     def gather_paths(prim: Usd.Prim):
          for child_prim in prim.GetAllChildren():
               if child_prim.GetTypeName() not in ["Mesh", "Shader", "Material", "Scope"]:
               # print(child_prim.GetPath().pathString, " -> ", child_prim.GetTypeName())
               robot_description.append(child_prim)
               gather_paths(child_prim)

     gather_paths(robot_prim)
     return robot_description

robot_prim = stage.GetPrimAtPath(robot_prim_path).GetParent()
robot_tree = get_robot_description(robot_prim)

The joints lists take in the name of the joints defined in the URDF. Then we loop through the robot's prims and find any PhysXRevoluteJoints and apply the appropriate API and properties. Any joint tuning you may have done by hand in the GUI can be added here provided you can find out the property name (there's a method for that in the comments).

The joint tuning guide is essential for this process. You can also look up the rigging robots section of the documentation.

from typing import List

velocity_driven_joints = ['wheel_left_joint', 'wheel_right_joint']
position_driven_joints = []
undriven_joints = []

# Set velocity, position, or undriven joints
for prim in robot_tree:
     prim_type = prim.GetTypeName()
     prim_path = prim.GetPath()
     if prim_type == "PhysicsRevoluteJoint":
          # print("At ", prim.GetPath().pathString, end=" ")
          prim_name = prim_path.pathString.split("/")[-1].strip()
        if prim_name in velocity_driven_joints:
            # print("-> Setting Velocity")
            drive = UsdPhysics.DriveAPI.Get(prim, "angular")
            drive.GetStiffnessAttr().Set(0)
            drive.GetDampingAttr().Set(1e10)
        elif prim_name in position_driven_joints:
            # print("-> Setting Position")
            drive = UsdPhysics.DriveAPI.Get(prim, "angular")
            drive.GetStiffnessAttr().Set(1e10)
            drive.GetDampingAttr().Set(0)
        else:
            # print("-> Setting Undriven")
            # print("Prop Names: ")
            # print(prim.GetPropertyNames())
            prim.RemoveAPI(UsdPhysics.DriveAPI, "angular")
            prim.GetProperty("physxJoint:jointFriction").Set(0.001)
            prim.GetProperty("physxJoint:maxJointVelocity").Set(4.0)

Running the script till here:

and the joint properties:

Attaching Sensors

This section covers attaching sensors to the robot. I’ll discuss publishing the data through ROS in the next post.

Lidars

We have 2 options:

PhysX Lidar → no intensity data: 255 for hit, 0 for no hit
RTX Lidar

RTX Lidar support is way better than PhysX, however clipping the max_angle of RTX Lidars was broken at the time of writing this post. There are workarounds for that but hopefully NVIDIA will fix it soon.

lidar_path: name of the lidar prim (can be anything)
lidar_parent: prim name of the robot's lidar link
lidar_config: pre-configure JSON files. Read more in the documentation

lidar_path = 'lidar'
lidar_parent = str(robot_prim.GetPrimPath()) + '/base_scan'

# RTX

# $ISAAC_PATH/exts/omni.isaac.sensor/data/lidar_configs for a list of other configs
lidar_config = "Example_Rotary"

_, sensor = omni.kit.commands.execute(
     "IsaacSensorCreateRtxLidar",
     path=lidar_path,
     parent=lidar_parent,
     config=lidar_config,
     orientation=Gf.Quatd(1.0, 0.0, 0.0, 0.0) # W, X, Y, Z
)

# PhysX
# _, _ = omni.kit.commands.execute(
#      "RangeSensorCreateLidar",
#      path=lidar_path,
#      parent=lidar_parent,
#      min_range=0.4,
#      max_range=30.0,
#      draw_points=False,
#      draw_lines=True, # set to True to visualize
#      horizontal_fov=360.0,
#      vertical_fov=30.0,
#      horizontal_resolution=0.4,
#      vertical_resolution=4.0,
#      rotation_rate=30.0,
#      high_lod=False,
#      yaw_offset=0.0,
#      enable_semantics=False
# )

Unfortunately I haven't played with sensor materials enough to write about them. If I do end up doing that I'll add it here. If you'd like to collaborate with me on that, contact me.

IMU

💡 Documentation

https://docs.omniverse.nvidia.com/isaacsim/latest/features/sensors_simulation/isaac_sim_sensors_physics_based_imu.html

imu_path = 'imu'
imu_parent = str(robot_prim.GetPrimPath()) + '/imu_link'

for prim in stage.TraverseAll():
    if prim.HasAPI(PhysxSchema.PhysxSceneAPI):
        current_physics_prim = prim
physx_scene_api = PhysxSchema.PhysxSceneAPI(current_physics_prim)
current_physics_frequency = physx_scene_api.GetTimeStepsPerSecondAttr().Get() # default = 60
dt = 1.0 / current_physics_frequency 

imu_result, imu_prim = omni.kit.commands.execute(
    "IsaacSensorCreateImuSensor",
    path=imu_path,
    parent=imu_parent,
    sensor_period=dt,
    visualize=False,
    linear_acceleration_filter_size=1,
    angular_velocity_filter_size=1,
    orientation_filter_size=1,
)

Camera

Default camera:

from omni.isaac.sensor import Camera

camera_path = str(robot_prim.GetPrimPath()) + '/camera_rgb_frame/camera'

camera_prim = Camera(
     prim_path=camera_path,
     frequency=20,
     resolution=(256, 256),
)

There are easy ways to get the calibration parameters for the cameras. Check out the full documentation here: https://docs.omniverse.nvidia.com/isaacsim/latest/features/sensors_simulation/isaac_sim_sensors_camera.html#calibrated-camera-sensors

Cameras seem to be the most loved sensor out of all. The documentation is really good so I’m not going to copy it here, but be sure to look up these files in the Isaac directory to learn more:

./standalone_examples/api/omni.isaac.sensor/camera_ros.py
./standalone_examples/api/omni.isaac.sensor/camera_opencv.py
./standalone_examples/api/omni.isaac.sensor/camera_opencv_fisheye.py

Wrap up

Run the script

./python.sh ./isaac-sim-starter/isaac.py

You should see the robot on a flat grid.

Create a new viewport for the camera: Window -> Viewport -> Viewport 2 and select camera

The next part will discuss how you can access the sensor data through ROS.

Isaac Sim Accelerated: Part 1

2024-07-01T00:00:00.000Z

Git Repository: https://github.com/ishandotsh/isaac-sim-starter

What to expect from this series

This is not an introduction to Isaac Sim. This series should be taken as one way you could set up a simulation environment for your robot. It can also act as a cheatsheet to copy useful snippets that could be tedious to write by hand (actiongraphs…). The reason I’m writing this article is because I wanted a post like this but couldn’t find one.

I recommend reading through the first 2 tutorials of the Isaac Sim Documentation to get a bearing.

P.S. I used ROS1 Noetic for this. I haven't used ROS2 yet so YMMV, but the basic principles of Isaac Sim will be the same.

Overarching Flow

The way I’ve worked it out is first you need to have your robot in a format that’s compatible with Isaac Sim. URDFs add a bit of complexity to the flow, but it is the most common robot description format. If you have the option, convert your asset to USD or USDA. You don’t need to go out of your way to learn about USDs because they’re a core part of the tool. Still, here’s a well curated list of resources.

After the robot’s loaded into an environment, you can add sensors to it. Adding sensors is very easy. The API is pretty clear and concise and I’ll get into it in the upcoming sections.

Once the robot is actually ready, its time for it to do tasks. This bit is very case specific so I’ll lay out the foundations and you’ll have to work out the rest. I’ll be happy to add your use case to this post and credit you.

Setting up the standalone script

Base Script

If you're following with the git repo set it up as follows:

cd ~/.local/share/ov/pkg/isaac_sim-2023.1.1 # might be different for you
git clone https://github.com/ishandotsh/isaac-sim-starter.git
cd isaac-sim-starter
./setup.sh
code . # open in your editor

In subsequent parts, I'll introduce several utility functions that make the base script cumbersome to read. Its best to create a separate file for those, so I'd suggest following the structure of the repo, and placing it inside the Isaac Sim install directory.

If instead you'd like to work on the app from a different directory, copy the .vscode directory and modify the setup.sh script. This is purely for python intellisense purposes. Then in a new python file:

from omni.isaac.kit import SimulationApp

app_config = {
     "width": "1280",
     "height": "720",
     "headless": False,
     "renderer": "RayTracedLighting",
}
# other config params:
# https://docs.omniverse.nvidia.com/py/isaacsim/source/extensions/omni.isaac.kit/docs/index.html

app = SimulationApp(app_config)

Load an environment

This will only work if you've set up the Nucleus server. Make sure that's always running before executing the script or you'll wait around forever for the sim to load.

from omni.isaac.core.utils.nucleus import get_assets_root_path
from omni.isaac.core.utils.stage import add_reference_to_stage

from omni.isaac.core import World
world = World(stage_units_in_meters=1.0)

assets_root_path = get_assets_root_path()

def spawn_environment(world_type: str = "ground_plane"):
     if world_type == "ground_plane":
          world.scene.add_default_ground_plane()
     elif world_type == "warehouse":
          warehouse_asset_path = assets_root_path + "/Isaac/Environments/Simple_Warehouse/warehouse.usd"
          warehouse_prim = add_reference_to_stage(usd_path=warehouse_asset_path, prim_path="/World/Warehouse")
     elif world_type == "warehouse_shelves":
          warehouse_asset_path = assets_root_path + "/Isaac/Environments/Simple_Warehouse/warehouse_multiple_shelves.usd"
          warehouse_prim = add_reference_to_stage(usd_path=warehouse_asset_path, prim_path="/World/Warehouse")
     elif world_type == "warehouse_full":
          warehouse_asset_path = assets_root_path + "/Isaac/Environments/Simple_Warehouse/full_warehouse.usd"
          warehouse_prim = add_reference_to_stage(usd_path=warehouse_asset_path, prim_path="/World/Warehouse")

# ground_plane, warehouse, warehouse_shelves, warehouse_full
spawn_environment("ground_plane")

# Other environments:
# https://docs.omniverse.nvidia.com/isaacsim/latest/features/environment_setup/assets/usd_assets_environments.html

Physics and lighting

Try setting solver type to PGS if TGS is running slow. More information here:

# https://openusd.org/release/api/index.html
from pxr import Gf, PhysxSchema, Sdf, UsdLux, UsdPhysics, Usd, UsdGeom
import omni.kit.commands

stage = omni.usd.get_context().get_stage()

# Physics 
scene = UsdPhysics.Scene.Define(stage, Sdf.Path("/physicsScene"))
scene.CreateGravityDirectionAttr().Set(Gf.Vec3f(0.0, 0.0, -1.0))
scene.CreateGravityMagnitudeAttr().Set(9.81)
PhysxSchema.PhysxSceneAPI.Apply(stage.GetPrimAtPath("/physicsScene"))
physxSceneAPI = PhysxSchema.PhysxSceneAPI.Get(stage, "/physicsScene")
physxSceneAPI.CreateEnableCCDAttr(True)
physxSceneAPI.CreateEnableStabilizationAttr(True)
physxSceneAPI.CreateEnableGPUDynamicsAttr(False)
physxSceneAPI.CreateBroadphaseTypeAttr("MBP")
physxSceneAPI.CreateSolverTypeAttr("TGS") # or PGS

# Lighting
distantLight = UsdLux.DistantLight.Define(stage, Sdf.Path("/DistantLight"))
distantLight.CreateIntensityAttr(500)

End of base script

# This part will always be at the end, all code from 
# the next steps should be above this bit

from omni.isaac.core import SimulationContext
simulation_context = SimulationContext()
simulation_context.play()

while app.is_running():
    try:
        simulation_context.step(render=True)

    except KeyboardInterrupt:
        break

simulation_context.stop()
app.close()

Run the base script

cd ~/.local/share/ov/pkg/isaac_sim-2023.1.1  # might be different for you
./python.sh ./isaac-sim-starter/isaac.py # or wherever you stored it

This should launch Isaac Sim and stay open until you ctrl-c in the terminal.

💡 More information about standalone scripts: https://docs.omniverse.nvidia.com/isaacsim/latest/introductory_tutorials/tutorial_intro_workflows.html#standalone-application

Toy Image Search Engine

2021-12-22T00:00:00.000Z

A search engine to find images containing the same object

In this project I used a pretrained ResNet34 model to get a feature map of a set of images. Then for a selected image, I get its feature vector with the model, and select images from the dataset with the lowest Euclidean distance. In this blog I'll explain how you can build something similar and the steps I followed to build it myself.

Step 1: The Dataset

In general, the larger the database, the better your search engine. I experimented with two datasets: ImageNet and the Pascal VOC datasets. Ultimately it doesn't matter which dataset you choose but I went with randomly picking ~1000 images from the Pascal VOC dataset because Heroku's free tier for app deployment restricts the size. In the project folder, place the dataset in a folder called data. Here's a random sample of the data:

Step 2: Generating and storing hidden features

I used feature vectors from an intermediate layer of PyTorch's pretrained ResNet34 model as a way to distinguish images.

Q. What are feature vectors?

They are lists of numbers to represent the image. In this case, they are the activations of the model's "neurons" of a particular layer when the image is used as input.

Q. Why not use the image itself as the feature vector?

There are a few reasons for this:

this would place too large an emphasis on the color values of the pixels of the image. This might be a good metric if we wanted to find images with similar colors but we want images with a similar object/situation
depending on the size of the image, the computation would be too heavy.

Let's get the basic part of the script out of the way. This will load the modules, set up the dataloader, and prepare the model to be used for inference:

# create_ftrs.py
# imports:
import torch
import torchvision
from torchvision import datasets, transforms
from torchvision.models import resnet34
from torch.utils.data import DataLoader, Dataset
import os
import numpy as np

# pytorch transforms to prepare for pretrained model
trfms = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    
    # this line is required for pretrained models in pytorch
    # find more details at https://pytorch.org/vision/stable/models.html
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

# define pytorch dataset and dataloader
data_dir = 'data'
# ImageFolder expects data in folders inside the root folder
dataset = datasets.ImageFolder(data_dir, trfms) 
# using a larger batch_size will speed everything up 
dl = DataLoader(dataset, batch_size=32, shuffle=False) 


# get pretrained model
model = resnet34(pretrained=True)
# send to GPU if you have it
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
# send eval mode on model since we're not training it
model.eval()

Now to extract the features, we need to register a "hook" to the model. This hook is just a function attached to a layer of the model that will run when its activations are calculated. I attached this hook to the model's avgpool layer.

# the feature vector gets saved to this dictionary
features = {}

def get_features(name):
    """
        returns the hook function itself
        the reason behind taking a name parameter is so if you're
        attaching multiple hooks, its easier to access them via a simple
        key lookup
    """
    def hook(model, input, output):
        features[name] = output.detach()
    return hook

# register the hook to the model
model.avgpool.register_forward_hook(get_features('data'))

The final step now that everything is set up is to run the model in inference mode and send the batches to it.

# The feature vectors of each batch get stored here
stored_ftrs = []

with torch.no_grad():
    # iterate through the dataloader
    for i, (img, label) in enumerate(dl):
        # print every 10 batches:
        if (i + 1) % 10 == 0:
            print(i+1)
        
        img = img.to(device)
        output = model(img)
        
        # append features['data'] to stored_ftrs
        # (remember the dictionary I created before)
        stored_ftrs.append(features['data'].cpu().numpy())

# finally, save the feature vectors array to file
np.save('stored_ftrs.npy', stored_ftrs, allow_pickle=True)

Step 3: The actual search

The generated feature vector of each image in the dataset is just that - a vector. So we can apply the concept of euclidean distance or L2 norm to find images that are close to a specific image.

Q. What is Euclidean distance / L2 norm?

Jason Brownlee on machinelearningmastery.com can explain this better I can

I defined some helper functions for this:

def compare(img):
    """ 
        does the exact same for the chosen image as we did
        for each image in the dataset
    """
    compare_ftrs = []
    with torch.no_grad():
        img = trfms(img)
        # unsqueeze is important because the model expects a batch of images
        img = img.unsqueeze(0)
        output = model(img)
        compare_ftrs.append(features['data'].cpu().numpy())
    return compare_ftrs

def get_distances(compare_img_tensor):
    # returns the distance of chosen image from each image in the dataset
    distances = []
    for batch in stored_ftrs:
        for img in batch:
            # L2 norm
            dist = np.linalg.norm(compare_img_tensor - img)
            distances.append(dist)
    return distances

def get_best_indices(distances, n=10):
    """
        Get the indices in the distances list of the n smallest 
        distances. 
    """
    dist_sorted = sorted(distances)
    image_idxs = []
    for d in dist_sorted[:n]:
        ind = distances.index(d)
        if ind not in image_idxs:
            image_idxs.append(ind)
    return image_idxs

def get_best_matches(image_idxs):
    """ 
        Returns the actual images from the dataset according to
        indices.
    """
    results = {}
    for i in image_idxs:
        results[i] = check_dataset.samples[i][0]
    return results

To put it all together, I will take an image, and run it through the helper functions. At the end I will get 10 similar images:

image = Image.open('santa_cat.jpg')
ftrs = compare(image)
distances = get_distances(ftrs[0])
best_indxs = get_best_indices(distances)
matches = get_best_matches(best_indxs)

# display (works jupyter notebooks):
for i in best_indxs:
    image = Image.open(matches[i])
    display(image)

Lets see if it works:

Input image:

Search results:

Looks like it works!

(optional) Step 4: Webapp

I used a popular framework called Streamlit to create a webapp for this simple search engine. Its pretty easy to do and you can check out the code on my Github or check out the live demo here (heroku free tier is dead)

I had to make some changes to make it suitable for deployment but they should not be too difficult to understand.

That's it!

That concludes the project! Thanks for reading it! Let me know if I've made any mistakes/missed anything/how I can make this better via my social links.