| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-09-09 | |||
| 19:05:45 | noonedeadpunk | ok, so `https://domain.com/console/vnc_lite.html?path=console%3Ftoken%3Dd7bf7a47-f081-4500-a961-05ddb7994e13` | |
| 19:05:48 | noonedeadpunk | does work indeed | |
| 19:07:05 | noonedeadpunk | so thanks a ton for your time | |
| 19:07:45 | noonedeadpunk | I know I was annoying but quite happy now as you said - it should be trivial both to rewrite and implement a "native" way | |
| 19:08:13 | sean-k-mooney | your bringing a pain point to our attention that not annoying | |
| 19:08:24 | sean-k-mooney | i still think subdomains are preferable | |
| 19:08:43 | sean-k-mooney | but if someone wants to add this minor feature im not agaisnt it | |
| 19:08:53 | sean-k-mooney | i just rpaobly wont go implemtnet it in my spare time | |
| 19:10:01 | sean-k-mooney | noonedeadpunk: you may have hit https://github.com/haproxy/haproxy/issues/829 by the way | |
| 19:12:11 | sean-k-mooney | there is a set-pathq to set items in teh query string now | |
| 19:14:13 | sean-k-mooney | i guess that migh now be https://www.haproxy.com/documentation/haproxy-configuration-tutorials/http-rewrites/#set-the-query-string | |
| 19:15:37 | sean-k-mooney | so i think you can jsut add "http-request set-query path=/subpath" | |
| 19:16:31 | sean-k-mooney | noonedeadpunk: im going to call it a day. if that works you could porpsoe a docs update if you were so inclidned or a devstack path | |
| 19:16:44 | sean-k-mooney | *patch | |
| 19:20:01 | noonedeadpunk | ++ thanks for the help as usual! | |
| 19:30:15 | noonedeadpunk | yeah, so here it goes: https://opendev.org/openstack/nova/src/branch/master/nova/objects/console_auth_token.py#L78 | |
| 19:49:53 | samcat116 | Hi all, whats a good place to start debugging the GET /servers api being slow? Running `openstack server list --all-projects --no-name-lookup` on a stack with about 1500 instances takes over 48s to return, with the GET https://example.com:8774/v2.1/servers/detail?deleted=False&all_tenants=True call taking over 30s itself. CPU usage across our 3 controllers is around 50% and I'm running with 10 api_workers, so don't think its a raw | |
| 19:49:54 | samcat116 | horsepower issue | |
| 22:10:54 | sean-k-mooney | samcat116: the server detail list need to hit neutron for each server to lookup secuirty groups, but openstack client is doing more then jsut /v2.1/servers/detail?deleted=False&all_tenants=Tr | |
| 22:10:57 | sean-k-mooney | | ue | |
| 22:12:40 | sean-k-mooney | that specific query however does require conenctin to all cell db and pulling a lot of info so its one of the more expensive api calls you can make to nova | |
| 22:13:05 | sean-k-mooney | its not as expensieve as simple tenant usage but it s more expensive then just about any other read only call | |
| 22:19:20 | samcat116 | Ok I guess I’ll look on the neutron side as we are pretty port and network heavy as well | |
| 22:19:37 | samcat116 | I just don’t know why it would be that slow | |
| 22:20:08 | sean-k-mooney | it might not be related to that but its on eof tthe peices of data we lookup form other services | |
| 22:20:23 | sean-k-mooney | we cache alotof the info reatled to other services but not all | |
| 22:20:56 | sean-k-mooney | tehre are some profiling tools liek os-profiler | |
| 22:21:06 | sean-k-mooney | but im not famialr with who to properly use them | |
| 22:21:45 | samcat116 | Ok. I do have memcache all setup so I would assume this could be heavily cached | |
| 22:22:01 | sean-k-mooney | no | |
| 22:22:17 | sean-k-mooney | i think we have almsot no caching for this | |
| 22:22:38 | sean-k-mooney | we use memcache very sparingly | |
| 22:23:31 | sean-k-mooney | its main use in nova is for the metadata api | |
| 22:24:15 | sean-k-mooney | we generally dont cache api responces | |
| 22:25:30 | sean-k-mooney | the other main use of memcafhe is for keytone auth | |
| 22:27:47 | samcat116 | Good to know | |
| 22:30:08 | samcat116 | How far off of an estimate could this be. Just trying to figure out if this is an order of magnitude slower than it should be or not | |
| 22:30:41 | sean-k-mooney | that kind of depend on other factors like how many cells you have | |
| 22:30:53 | sean-k-mooney | you not the first person to ask about this | |
| 22:31:13 | sean-k-mooney | samcat116: can you try making the same query to /server instead of /server/detials | |
| 22:31:47 | sean-k-mooney | the delta is /server just hits the api db and list all the server we know about without going to the cell dbs or calling other services | |
| 22:32:17 | sean-k-mooney | that should be very fast as it basicaly a select instance_uuid from istance_mappings | |
| 22:35:01 | sean-k-mooney | if /servers/ is fast and /servers/detail is slow then one thing to check is if any of the cells are slow to respond | |
| 22:35:31 | sean-k-mooney | internally the api when repondign ot /servers/detail is calling into each cell db and geting the data form the instances table | |
| 22:38:06 | sean-k-mooney | the security group info i thin need a call to neutron for every port | |
| 22:38:41 | samcat116 | Ok I’ll try that. We only have one cell | |
| 22:39:09 | sean-k-mooney | that should simplfy thing then as the api is not making parralel requests | |
| 22:41:07 | sean-k-mooney | i think the security gorups are the only thing not cached in that responce. i.e. that is not just a direct db lookup | |
| 22:42:39 | sean-k-mooney | this has been know for a long time https://bugs.launchpad.net/nova/+bug/1923560 and a fix was prospoed here https://review.opendev.org/c/openstack/nova/+/786348 | |
| 22:43:39 | sean-k-mooney | i was personaly on the side of removing the security groups form the instance obejct entirly | |
| 22:43:46 | sean-k-mooney | btu if we dont then we shoudl cache them | |
| #openstack-nova - 2024-09-10 | |||
| 00:04:27 | samcat116 | So /server?all_tenants=True vs /servers/detail?all_tenants=True was 1.3s vs 26s | |
| 00:04:33 | samcat116 | s/server/servers/, s/all_tenants/all\_tenants/, s/all_tenants/all\_tenants/ | |
| 00:18:06 | sean-k-mooney | we woudl expect there to be a delta but not over an order of magnitde. | |
| 07:04:49 | bauzas | because we don't call the cells DB for the first call :) | |
| 07:08:37 | opendevreview | Tobias Urdin proposed openstack/nova master: api-ref: Add new lines not allowed for server description https://review.opendev.org/c/openstack/nova/+/928758 | |
| 09:06:50 | bauzas | well, looking at the relnotes, the cycle highlights will be quite simple to write... | |
| 09:26:29 | opendevreview | Balazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property https://review.opendev.org/c/openstack/nova/+/928456 | |
| 09:26:30 | opendevreview | Balazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb https://review.opendev.org/c/openstack/nova/+/928584 | |
| 09:33:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb https://review.opendev.org/c/openstack/nova/+/928584 | |
| 09:39:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Refactor obj_make_compatible to reduce complexity https://review.opendev.org/c/openstack/nova/+/928590 | |
| 09:39:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property https://review.opendev.org/c/openstack/nova/+/928456 | |
| 09:39:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb https://review.opendev.org/c/openstack/nova/+/928584 | |
| 09:46:54 | opendevreview | Balazs Gibizer proposed openstack/os-traits master: Add support for hw_vif_model igb image property https://review.opendev.org/c/openstack/os-traits/+/928582 | |
| 11:59:23 | sean-k-mooney | gibi: for the igb support can you allso add a patch to glance to update https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L112-L127 | |
| 11:59:41 | sean-k-mooney | that used to autogenerate the ui in glance/heat ectra | |
| 12:01:01 | sean-k-mooney | i was going to set that via horizon and rememberd that its defiend as a enum so you cant do that without the metadef update | |
| 12:03:21 | sean-k-mooney | gibi: setting it via the command line works prefectly fine | |
| 12:04:51 | sean-k-mooney | one slight problem is that igb is not included in the cirros image i belive | |
| 12:06:55 | sean-k-mooney | ya ok its not | |
| 12:07:29 | sean-k-mooney | that wont be a problem for us when trying to enable this in ci for the host | |
| 12:07:39 | sean-k-mooney | but we may want ot patch cirros to include the module going forward | |
| 12:07:51 | sean-k-mooney | frickler: ^ that would be nice to have in cirros 7 | |
| 12:14:23 | frickler | sean-k-mooney: what's the advantage of igb? I think I'm lacking some context | |
| 12:14:45 | sean-k-mooney | frickler: it support sriov in the guest i.e. you can create 7 vfs | |
| 12:15:14 | sean-k-mooney | we want to enable it in nova so we can eventually use it for the host vms to do sriov testing in the first party ci | |
| 12:15:55 | sean-k-mooney | qemu virtualises the sriov supprot for igb so there is no hardware dep provided your qemu/libvirt is new enough of course to supprot igb | |
| 12:16:09 | sean-k-mooney | ubuntu 24.04 ships new enough versions fo qemu for this | |
| 12:17:08 | gibi | sean-k-mooney: yeah cirros does not have the igb kernel module available ubuntu has it by default | |
| 12:17:29 | gibi | sean-k-mooney: sure I will look at the glance update | |
| 12:18:06 | sean-k-mooney | its one line so pretty simple. ya im uploading a ubuntu image now | |
| 12:18:23 | sean-k-mooney | lol ok i forgot ther .img images are qcow and it failed the format check | |
| 12:18:50 | frickler | yes, cirros reduces the number of modules available to some useful ones, most you don't need within a vm. including all that ubuntu offers would almost double the cirros image size | |
| 12:19:36 | sean-k-mooney | ya so we have e1000 aviabel just not igb | |
| 12:19:54 | sean-k-mooney | igb is the intel 1g nich that replaced e1000 | |
| 12:20:05 | sean-k-mooney | but its still 10 years old at this point if not more | |
| 12:20:22 | sean-k-mooney | igb was the first generation to support sriov on the intel side | |
| 12:23:51 | gibi | sean-k-mooney: done https://review.opendev.org/c/openstack/glance/+/928788 | |
| 12:24:32 | gibi | frickler: yeah I don't ask to include all the modules, just igb :) | |
| 12:24:50 | gibi | and igbvf | |
| 12:25:11 | sean-k-mooney | oh right the vf driver for using a vf in the guest | |
| 12:25:17 | sean-k-mooney | yep that would be needed too | |
| 12:34:36 | sean-k-mooney | you know its alwasy the little things.. like uploading a keypair before booting a ubuntu vm... | |
| 12:36:04 | sean-k-mooney | i can see its probing proeprly in the guest however form the console logs | |
| 12:36:05 | sean-k-mooney | [ 110.003524] igb 0000:00:03.0 ens3: renamed from eth0 | |
| 12:36:40 | sean-k-mooney | and its set properly in the xml | |
| 12:36:55 | sean-k-mooney | gibi: so at first glance this all looks like tis workign properly | |
| 12:37:24 | sean-k-mooney | gibi: ill review this properly once we have approved the bluepint and do some more testing at that point | |
| 12:40:07 | sean-k-mooney | but it is functional form what i can tell as is. | |
| 12:40:26 | sean-k-mooney | i dont have the patched os-traits in use here so the trait was not set in placement | |