| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-07 | |||
| 16:19:36 | tomtom001 | sean-k-mooney thank you, lyarwood, you have anything I can check or something you'd like to look at to tell you? | |
| 16:42:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send rp_uuid in the port binding https://review.openstack.org/569459 | |
| 16:42:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | |
| 16:42:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 16:42:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations to neturonv2 api https://review.openstack.org/616240 | |
| 16:50:21 | efried | mriedem: Update on the nova-next business... | |
| 16:50:41 | efried | https://review.openstack.org/#/c/616033/ see latest comment | |
| 16:50:58 | efried | tldr I don't think the tweak is working, or I'm a dolt and looking in the wrong place. | |
| 16:52:34 | efried | However, having looked at the placement logs for the nova-next runs in the whole series, I see a major drop at the "Reduce calls to placement from _ensure" patch, which is what I would expect; and actually, I wouldn't expect any of the CI jobs to see a significant drop from setting refresh=0 because they're getting refreshed all the time because they're going crazy doing real stuffs, so the refreshing is happening anyway and the | |
| 16:53:25 | efried | The refresh=0 thing is really only going to see a big benefit in steady state, and (afaik) we don't have any CI jobs that just... sit there. | |
| 16:54:57 | mriedem | i've had issues with using post-config and NOVA_CPU_CONF in the nova-live-migration job... | |
| 16:56:12 | efried | mriedem: I found a cells job that was using something similar and it seemed to be working (was able to verify the conf change was made) | |
| 16:56:54 | efried | mriedem: http://logs.openstack.org/33/616033/1/check/nova-cells-v1/fb84e57/logs/etc/nova/nova-cells.conf.txt.gz is getting vif_plugging_timeout = 0 from playbooks/legacy/nova-cells-v1/run.yaml | |
| 16:58:08 | mriedem | http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_24_44_044 | |
| 16:58:08 | mriedem | devstack is processing the post-config | |
| 17:00:19 | efried | so... wtf? | |
| 17:00:57 | mriedem | it looks like that post-config happens first, | |
| 17:01:03 | mriedem | and then other stuff is writing to nova-cpu.conf | |
| 17:01:13 | mriedem | so my guess is that devstack is overwriting it... | |
| 17:01:51 | efried | ah http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_26_29_702 | |
| 17:01:55 | efried | we blow it away later :( | |
| 17:02:09 | efried | what's the magic env var for the nova.conf? | |
| 17:02:15 | mriedem | yup http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_26_29_702 | |
| 17:02:24 | mriedem | $NOVA_CONF | |
| 17:03:37 | mriedem | definitely a bug, but just do $NOVA_CONF for now and comment in the playbook | |
| 17:03:44 | efried | ack | |
| 17:04:04 | mriedem | i can report a devstack back | |
| 17:04:14 | efried | bug? | |
| 17:04:16 | efried | cool, thanks. | |
| 17:06:20 | openstack | Launchpad bug 1802143 in devstack "Changes from [[post-config|$NOVA_CPU_CONF]] are overwritten in "superconductor" mode" [Undecided,New] | |
| 17:06:20 | mriedem | https://bugs.launchpad.net/devstack/+bug/1802143 | |
| 17:06:21 | mriedem | *yeah bug | |
| 17:07:35 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Turn off rp association refresh in nova-next https://review.openstack.org/616033 | |
| 17:07:41 | efried | mriedem: done ^ | |
| 17:08:07 | efried | mriedem: but as I said, I don't actually expect to see a significant difference, because active state. | |
| 17:08:28 | efried | though making sure it actually works and doesn't blow up the world will be good :) | |
| 17:10:43 | mriedem | hey now that the dems took over the house the world is going to end anyway | |
| 17:11:30 | mriedem | heh, *braces | |
| 17:11:50 | mriedem | sorry, trump is doing a news conference talking about the lack of republican losers that didn't accept his warm embrace | |
| 17:25:11 | tssurya | sean-k-mooney: I had a question, would you know the working of virsh capabilities (ca you point me to the code of how this xml is generated)? | |
| 17:25:22 | tssurya | couldn't find the exact source code | |
| 17:27:26 | mriedem | kashyap: ^ | |
| 17:29:27 | dansmith | tssurya: it comes from libvirt, obviously | |
| 17:29:36 | dansmith | tssurya: there is a lot of documentation about what they all mean | |
| 17:29:49 | tssurya | dansmith: yea I tried to go into the libvirt code,, but in vain | |
| 17:29:55 | dansmith | tssurya: https://libvirt.org/formatcaps.html | |
| 17:30:06 | dansmith | tssurya: you want to know how it's determining one thing or another, or just what everything means? | |
| 17:30:09 | tssurya | I already saw that link | |
| 17:30:35 | dansmith | example of what you're trying to figure out? | |
| 17:30:49 | tssurya | I want to know how this Core(s) per socket in the topology is calculated | |
| 17:31:04 | tssurya | or the code in libvirt that gets this part | |
| 17:31:58 | dansmith | it should all be straight out of /sys I would think | |
| 17:32:25 | dansmith | you're talking about host caps in that case right? not how a guest's topo gets determined (by nova) but how libvirt reports the host topo? | |
| 17:32:49 | tssurya | yea the host caps (I think) | |
| 17:32:52 | sean-k-mooney | tssurya: if you talking about the guest topology the answer is dumbly | |
| 17:33:25 | tssurya | basically we had a live migration failure because the cpu_info didn't match | |
| 17:33:54 | dansmith | there are lots of reasons for that being possible, topo is one and flags being another | |
| 17:34:06 | tssurya | which we started investigating and reached the topo part | |
| 17:37:57 | sean-k-mooney | tssurya: so ya as dansmith mentioned libvirt gets this info from sysfs which the bios/uefi proved to the kernel based on probing the cpu. the kenrel can re probe this info if it really needs to but libvirt is just taking whatever the kernel provides | |
| 17:38:34 | tssurya | oh okay | |
| 17:39:07 | sean-k-mooney | tssurya: the toplogy info is here ls /sys/bus/cpu/devices/cpu0/topology/ | |
| 17:39:13 | sean-k-mooney | the cpu flags are up one level | |
| 17:39:51 | dansmith | sean-k-mooney: live migration failing because of topology would only be if you're using pinned cpus and those cpus don't exist on the remote side right | |
| 17:39:52 | dansmith | ? | |
| 17:39:55 | tssurya | thanks, looking | |
| 17:40:15 | sean-k-mooney | dansmith: yes | |
| 17:40:51 | tssurya | yea we have something like 48 cores source out of which 24 are offline and 24 on destination, but the 24 offline is not reflected or something | |
| 17:40:53 | openstackgerrit | Jack Ding proposed openstack/nova master: Add HPET timer support for x86 guests https://review.openstack.org/605902 | |
| 17:41:01 | tssurya | not sure if that makes sense | |
| 17:41:36 | sean-k-mooney | tssurya: have you excluded the offline cpus form the vcpu_pin_set | |
| 17:42:17 | sean-k-mooney | tssurya: use because they are offline does not mean they will not be in sys / virsh capablities | |
| 17:42:51 | sean-k-mooney | tssurya: but ya as dan said topology will only come into it if the cpus you were pinned too on the source are not present on the dest | |
| 17:43:14 | sean-k-mooney | tssurya: if they are present it could still be incorrect to do the migration but that is a sperate issue | |
| 18:05:53 | cfriesen | tssurya: as I'm sure you're aware, live migration with CPU pinning is currently broken. :) | |
| 18:06:50 | tssurya | sean-k-mooney: how do I exclude offline cpus from vcpu_pin_set (sorry I am a totally beginner in this stuff) | |
| 18:06:58 | tssurya | cfriesen: actually I wasn't :) | |
| 18:07:03 | tssurya | thanks for telling me | |
| 18:08:07 | tssurya | sean-k-mooney: because we were looking through the libvirt code and like you said it seems to be looping through all the cores even if they are offline | |
| 18:08:32 | cfriesen | tssurya: technically it's never been reliable. it will try to keep the same topology and host cpus on the dest node, even if those cpus/numa-nodes don't exist or are already in use by other instances. | |
| 18:09:40 | cfriesen | tssurya: there's work in progress to fix it finally by recalculating the resources on the dest as part of the live migration, so that might land in stein | |
| 18:10:16 | tssurya | cfriesen: ah good to know | |
| 18:25:12 | openstackgerrit | sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034 | |
| 18:25:31 | sean-k-mooney | gibi: ^ should have the exception behavior you wanted | |
| 18:29:28 | sean-k-mooney | gibi: the run_once contract is now expcitly stated too. the contract is different then i had planned but it will still work for my usecase and hopefully you find it more intuititve | |
| 18:34:05 | cdent | edleafe: if you get a chance to cast your eye on https://review.openstack.org/#/c/603955/ (cyborg interaction spec) I think it could do with it. The API bits need some stronger review and I've run out of time and I believe you've got some state on the cyborg stuff? | |
| 18:36:35 | sean-k-mooney | cdent: do you think the spec is close | |
| 18:37:17 | sean-k-mooney | cdent: i spend a lot of time going through it in detail initilly but i have not had time to go through the spec end to end in several interations | |
| 18:40:45 | cdent | sean-k-mooney: it's hard to say. a lot of the details are kind of lost on me. When I previously reveiwed it was mostly to look at the shape of the API, which I found a bit off (which is why I've asked edleafe to look as a fellow api-sig person). | |
| 18:42:21 | edleafe | cdent: It's on my plate. I've had the same experience as sean-k-mooney - it's hard to get through the entire thing | |
| 18:42:54 | sean-k-mooney | cdent: i have had similar feeling. i have tried to blance the "is the cyborg api correct" comments with is the workflow and datastuctre used correct in terms of the nova interaction | |
| 18:44:08 | sean-k-mooney | edleafe: there is still alot going on for a single spec but some of that context/complexity is unaviodable unfortunetly | |
| 18:44:24 | edleafe | sean-k-mooney: agreed | |
| 18:48:53 | artom | mriedem, oh hai, could you hit https://review.openstack.org/#/c/587248/1 and the one on top when you get a chance? It's the pike backport of a thing you +2'ed in queens | |
| 18:56:32 | mriedem | ok | |
| 19:00:19 | mriedem | done | |
| 19:12:21 | artom | Cheers! | |
| 19:27:12 | openstack | Launchpad bug 1802167 in OpenStack Compute (nova) "nova-manage db purge -h has weird help for --before option" [Low,Triaged] | |
| 19:27:12 | mriedem | super critical bug that needs fixing asap https://bugs.launchpad.net/nova/+bug/1802167 | |
| 19:27:44 | artom | oh shiite im on that | |
| 19:27:59 | sean-k-mooney | artom: feels more like a stephenfin thing :) | |