| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-09 | |||
| 21:20:39 | sean-k-mooney | sure but not in a patch seperate form the new usage or the full object | |
| 21:22:21 | sean-k-mooney | artom: sorry this just annoys me because you were chanign someint i did not think you were going to change. it conflicts with my changes and it break my mental model of how the pci trakcer works | |
| 21:23:34 | artom | sean-k-mooney, that changes aren't that dramatic... | |
| 21:24:47 | sean-k-mooney | the main one was that it never store state in the pci tracker object directly | |
| 21:25:41 | sean-k-mooney | the numa toplogy object will have to be keep consitent now | |
| 21:26:23 | sean-k-mooney | ok we do sotre state but differently | |
| 21:26:54 | artom | We're talking about the *host* numa_topology | |
| 21:26:59 | sean-k-mooney | yes | |
| 21:27:00 | artom | When is that ever going to change... | |
| 21:27:57 | sean-k-mooney | within th elife time of the agent i guess it not going to | |
| 21:28:24 | sean-k-mooney | i mean memoy and cpu hotplug are thigns and you can reconfigure hyperthreading on the fly or hugepagers for that matter | |
| 21:28:40 | sean-k-mooney | hugepage is actully the most likely ot change at runtime | |
| 21:28:52 | sean-k-mooney | but to have that picked up you need to restart libvirtd | |
| 21:29:51 | sean-k-mooney | actully se also store the currently pinne cpu in the host numa toplogy blob | |
| 21:30:22 | sean-k-mooney | so its update every time we boot new vms | |
| 21:36:28 | artom | sean-k-mooney, the PCI tracker never uses that information though | |
| 21:36:45 | artom | I could add a comment to warn future programmers | |
| 21:37:08 | artom | We just need the socket/node mapping, and that's effectively constant | |
| 21:40:11 | lyarwood | melwitt / sean-k-mooney ; so in my defence, it's late and I shouldn't be working but when I said ~5mins earlier what I actually wanted to say was ~500 seconds. http://paste.openstack.org/show/803406/ | |
| 21:41:00 | sean-k-mooney | lyarwood: ay right that about right with paralle | |
| 21:42:01 | sean-k-mooney | i think i missed where you said ti too ~500 though | |
| 21:44:00 | sean-k-mooney | sub ten mintues is doable if you have good netowrking, io and a fast cpu | |
| 21:46:00 | lyarwood | yup I'm just running a 4 vCPU, 16GB, 50GB RAW disk VM on my p1 gen2 with a 1Gbps connection | |
| 21:52:11 | melwitt | lyarwood: heh, sorry, it wasn't only you, I had seen other mentions of 5-6 min about it before and your mention made me think to ask what am I doing wrong to not get this result 😆 | |
| 21:52:49 | sean-k-mooney | melwitt: yep dansmith was around the 5-8 minute mark | |
| 21:53:06 | sean-k-mooney | that i think was on baremetal | |
| 21:53:25 | sean-k-mooney | rather then nested virt but its doable | |
| 21:54:15 | dansmith | talking about devstack time? | |
| 21:54:22 | sean-k-mooney | yep | |
| 21:55:27 | sean-k-mooney | i think you were geting about 430 ish second if im not mistaken | |
| 21:55:30 | dansmith | yeah, I can do about 5mins with a less-than-full devstack config | |
| 21:55:49 | dansmith | with OCaaS plus parallel I can get 3xxx yeah | |
| 21:56:00 | dansmith | er, 3xx | |
| 21:56:11 | melwitt | what are the main things you disable? | |
| 21:56:26 | dansmith | disable_service c-bak etcd3 c-api c-vol c-sch swift horizon dstat | |
| 21:56:34 | melwitt | thanks | |
| 21:56:34 | dansmith | tempest if I don't need it | |
| 21:56:51 | sean-k-mooney | swift i think is kind of slow to set up | |
| 21:57:18 | sean-k-mooney | dstat should not make much of a differnce horizon take a while to complie and compress the static pages | |
| 21:57:52 | dansmith | well, not all of the systemctl commands are super fast | |
| 21:58:17 | dansmith | sometimes depending on what is running, daemon-reload can take a couple seconds, and start if it waits for the first child, etc | |
| 22:00:19 | sean-k-mooney | ya i notice that more on unstack then anything esle | |
| 22:00:45 | sean-k-mooney | some service take a long time to stop randomly | |
| 22:00:46 | dansmith | yeah, that's another good reason though, | |
| 22:01:01 | dansmith | more shtuff to unstack makes the process slower when you're iterating | |
| 22:02:10 | sean-k-mooney | yep although i normlaly see how long i can go with just doing sudo systemctl restart devstack@n-* | |
| 22:02:43 | sean-k-mooney | if im hacking on stuff i generally dont restack unless i have too | |
| 22:04:17 | dansmith | well, when you're working on stuff that crosses multiple projects, as I have been lately, unstack/stack time is important | |
| 22:04:25 | dansmith | especially if one of those _is_ devstack :) | |
| 22:04:49 | sean-k-mooney | yep i used to restack multiple times a day | |
| 22:05:14 | sean-k-mooney | now i just have different envs for different tings | |
| 22:05:19 | sean-k-mooney | so i restack less | |
| 22:05:44 | sean-k-mooney | still important to be quick | |
| 22:06:20 | dansmith | yeah just depends on what you're doing | |
| 22:06:30 | dansmith | obviously hacking on a single project, service restart is by far the most efficient :) | |
| 22:07:14 | sean-k-mooney | basically if i dont need db change i try to jsut checkout the patch i need and restart it | |
| 22:07:22 | sean-k-mooney | if i get error i restack | |
| 22:07:39 | sean-k-mooney | it works more times then it proably should | |
| 22:09:42 | dansmith | it *should* work for most things, so .. I'd be concerned if it didn't ;) | |
| 22:10:25 | sean-k-mooney | i sometime get bitten by the compute service verion if i change to a different series that i want to test | |
| 22:10:41 | sean-k-mooney | i know i can fix that but i never do | |
| 22:10:47 | sean-k-mooney | i just restack | |
| #openstack-nova - 2021-03-10 | |||
| 02:33:39 | openstackgerrit | Jinsheng Zhang proposed openstack/nova-specs master: Add nova support ironic instance port group network metadata spec https://review.opendev.org/c/openstack/nova-specs/+/779644 | |
| 07:54:45 | openstackgerrit | Merged openstack/nova stable/ussuri: Default user_id when not specified in check_num_instances_quota https://review.opendev.org/c/openstack/nova/+/777217 | |
| 07:55:03 | openstackgerrit | Merged openstack/nova stable/ussuri: Add regression test for bug 1914777 https://review.opendev.org/c/openstack/nova/+/777218 | |
| 07:55:05 | openstack | bug 1914777 in OpenStack Compute (nova) victoria "Possible race condition between n-cpu and n-api when deleting a building instance" [High,In progress] https://launchpad.net/bugs/1914777 - Assigned to melanie witt (melwitt) | |
| 08:02:00 | openstackgerrit | Merged openstack/nova stable/ussuri: Handle instance = None in _local_delete_cleanup https://review.opendev.org/c/openstack/nova/+/777219 | |
| 08:09:00 | gibi | good morning | |
| 08:27:14 | yonglihe | good morning | |
| 08:40:44 | openstackgerrit | Yongli He proposed openstack/nova master: Smartnic support - cyborg drive https://review.opendev.org/c/openstack/nova/+/771362 | |
| 08:40:46 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support - new vnic type https://review.opendev.org/c/openstack/nova/+/771363 | |
| 08:40:48 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 08:47:49 | openstackgerrit | Adit Sarfaty proposed openstack/nova master: Retry on vmware create_vm when it fails https://review.opendev.org/c/openstack/nova/+/764586 | |
| 09:20:56 | bauzas | good morning | |
| 09:25:00 | yonglihe | sean-k-mooney: refer to patch comments, really hope that address those concerns. for first patch commit message changed thanks. | |
| 09:31:13 | gibi | bauzas: jobs are runnig this morning as far as I see | |
| 09:36:56 | bauzas | gibi: I haven't seen jobs running in any sbg 1 to 4 dcs | |
| 09:37:16 | bauzas | but /me gives hugs to OVH folks around | |
| 09:37:45 | bauzas | my backups are impacted, but meh ;) | |
| 10:15:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove dead error handling code https://review.opendev.org/c/openstack/nova/+/779704 | |
| 10:19:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/778885 | |
| 10:32:40 | stephenfin | kashyap: Can you take a look at https://review.opendev.org/c/openstack/nova/+/779304/ today, please? | |
| 10:36:06 | stephenfin | lyarwood: Seen this before? https://zuul.opendev.org/t/openstack/build/a19ff22927a045df9f8982d7ef678238/log/controller/logs/screen-n-cpu.txt#16367 | |
| 10:36:41 | stephenfin | WARNING: Failed to get udev device handler for device /dev/sda1.\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n ... | |
| 10:38:03 | stephenfin | artom: Look what I see in some build logs -> This host appears to have multiple sockets per NUMA node. The `socket` PCI NUMA affinity will not be supported. | |
| 10:38:10 | stephenfin | artom: from https://zuul.opendev.org/t/openstack/build/a19ff22927a045df9f8982d7ef678238/log/controller/logs/screen-n-cpu.txt | |
| 10:42:36 | lyarwood | stephenfin: yeah ./me finds the bug | |
| 10:43:01 | lyarwood | stephenfin: https://bugs.launchpad.net/cinder/+bug/1901783 | |
| 10:43:03 | openstack | Launchpad bug 1901783 in Cinder "volume delete fails because cinder-rootwrap lvs fails with exit code 139" [Undecided,Triaged] | |
| 10:44:09 | stephenfin | oh, fun. This one is happening in nova (via privsep) | |
| 10:44:23 | lyarwood | oh really? | |
| 10:44:32 | lyarwood | sorry I didn't actually click through | |
| 10:44:37 | stephenfin | all good | |
| 10:44:37 | stephenfin | https://a59f59e90e7507f14599-2cecd17ff9ea5f4d02c12f6d5fc5aedd.ssl.cf1.rackcdn.com/776681/7/check/nova-lvm/a19ff22/controller/logs/screen-n-cpu.txt | |
| 10:44:44 | stephenfin | search for 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43 | |
| 10:44:44 | lyarwood | ah right it's the nova-lvm job | |
| 10:44:51 | lyarwood | that makes sense | |
| 10:44:51 | stephenfin | (the instance) | |
| 10:45:02 | stephenfin | yup, agreed | |
| 10:47:18 | kashyap | stephenfin: Mornin, I have it open; will definitely look today. (Just finishing something more time-constrained.) | |