| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-25 | |||
| 16:18:25 | artom | Sounds like we need the equivalent of a carbon tax | |
| 16:18:28 | efried | big difference between obsolete nova code and something that was never relevant to nova. | |
| 16:19:46 | stephenfin | relevant to nova developers though | |
| 16:20:03 | stephenfin | who are as important, if not more important, than the code | |
| 16:20:12 | efried | as relevant to any project's developers, nah? | |
| 16:20:43 | efried | are you going on a crusade to propose this same thing to all projects you work on? | |
| 16:21:08 | artom | cdent, woot, I'm back to last night's failure | |
| 16:21:19 | cdent | something reasonable now | |
| 16:21:20 | artom | *tackling | |
| 16:21:35 | stephenfin | I'll probably add it to one or two of my personal projects, maybe Sphinx too, but I wouldn't be touching oslo and the likes, no | |
| 16:21:37 | kashyap | artom: Hehe, one letter changes the meaning, doesn't it :D | |
| 16:21:59 | stephenfin | Basically anywhere where I'm likely to be undertaking large feature work consisting of many patches | |
| 16:22:23 | artom | kashyap, at least I wasn't tickling it | |
| 16:23:15 | kashyap | mriedem: Try this, not sure if that's your glass of (root?) beer -- https://www.youtube.com/watch?v=jBo870lVUyc | |
| 16:23:57 | kashyap | [Preferably with a good quality headset / speaker] | |
| 16:27:26 | cdent | oh. that's nice. | |
| 16:29:15 | kashyap | Jimmy Smith++ | |
| 17:02:46 | mriedem | what in tarnations, created devstack from master today, create a server, n-cpu logs say the guest was created in the hypervisor, and then things just hang - and virsh list doesn't show anything | |
| 17:02:47 | mriedem | wtf | |
| 17:05:36 | sean-k-mooney | mriedem: im guessing libvirt crahsed | |
| 17:05:57 | sean-k-mooney | either that or you need to run virsh listh with either sudo or --all | |
| 17:06:19 | sean-k-mooney | actully if it hung then ingore the last bit | |
| 17:06:26 | mriedem | oh right sudo virsh list | |
| 17:06:31 | mriedem | libvirtd is green | |
| 17:06:46 | mriedem | the domain is just hung in paused state | |
| 17:07:27 | sean-k-mooney | the domain or the nova compute agent | |
| 17:07:38 | mriedem | the domain | |
| 17:07:47 | mriedem | 3 instance-00000003 paused | |
| 17:07:47 | mriedem | ---------------------------------------------------- | |
| 17:07:47 | mriedem | Id Name State | |
| 17:07:47 | mriedem | $ sudo virsh list --all | |
| 17:08:03 | sean-k-mooney | and in nova its active | |
| 17:08:09 | mriedem | no | |
| 17:08:20 | mriedem | it's building b/c the libvirt driver is waiting for the power state to change from paused to running | |
| 17:09:30 | sean-k-mooney | oh and you it alwready told it to unpause? we start the domian in the pased state. i wonder if the qemu monitor has hung | |
| 17:14:14 | mriedem | Jul 25 17:13:36 devstack systemd[1]: libvirt-guests.service: Control process exited, code=exited status=1 | |
| 17:14:14 | mriedem | Jul 25 17:13:36 devstack libvirt-guests.sh[18879]: Timeout expired while shutting down domains | |
| 17:14:14 | mriedem | fun | |
| 17:14:21 | mriedem | trying to restart libvirt-guests | |
| 17:15:19 | sean-k-mooney | is this a clean install of ubuntu 18.04? | |
| 17:15:37 | mriedem | well from a vexxhost image of 18.04 but yeah | |
| 17:16:15 | sean-k-mooney | strange i personlly havent had any issue with 18.04 i did an install fiday | |
| 17:16:33 | mriedem | me neither | |
| 17:17:57 | sean-k-mooney | are the vexhost image available for download | |
| 17:18:06 | mriedem | idk, i'm trashing this vm | |
| 17:19:12 | sean-k-mooney | ya i would just start over too to be honest. i suspect its somehitng to doe with libvirt/qemu or maybe apparmor but i would start clean | |
| 17:19:12 | mnaser | the vexxhost images are straight up the ones shipped by ubuntu | |
| 17:19:27 | sean-k-mooney | mnaser: the cloud images | |
| 17:19:32 | mnaser | yep | |
| 17:19:38 | mnaser | only thign we do is convert from qemu to raw | |
| 17:19:41 | mnaser | and upload | |
| 17:19:53 | sean-k-mooney | im guessing ye are using ceph as a backend then | |
| 17:20:12 | mnaser | indeed :) | |
| 17:20:13 | kashyap | On Fedora, the 'libvirt-guests' thing isn't even enabled: | |
| 17:20:15 | kashyap | Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; disabled; vendor preset: disabled) | |
| 17:20:15 | kashyap | ● libvirt-guests.service - Suspend/Resume Running libvirt Guests | |
| 17:20:15 | kashyap | $> systemctl status libvirt-guests | |
| 17:20:18 | kashyap | Active: inactive (dead) | |
| 17:20:21 | kashyap | ... | |
| 17:20:46 | kashyap | But yeah, that timeout of 'libvirt-guests' looks spurious enough, might as well start over. | |
| 17:21:05 | sean-k-mooney | kashyap: would that not cause filesystem curruption if you did not suspend them on rebooting the host | |
| 17:21:06 | kashyap | (Also, not sure if that paused instance's QEMU process went 'defunct') | |
| 17:21:37 | efried | following up re os-vif and python-novaclient releases: Libs are required to do one release per milestone. os-vif was last released at m1, so we can expect the release team to propose that one. python-novaclient was released a couple weeks ago, so we're probably good on that one. | |
| 17:21:41 | sean-k-mooney | kashyap: im guessing the qemu moniotr process stoped processing messages form libvirt | |
| 17:22:47 | sean-k-mooney | efried: ok there is one think i would like to fix soonish but im only starting on it today | |
| 17:22:48 | kashyap | Yeah, but that doesn't tell us why. It could be any no. of reasons | |
| 17:23:33 | sean-k-mooney | kashyap: yep its proably quicker to kill it and spin up a clean vm | |
| 17:23:45 | sean-k-mooney | if mriedem hits it again we can take another look | |
| 17:24:22 | kashyap | sean-k-mooney: Yeah, on FS corruption, possibly "enterprise distros" would enable it | |
| 17:24:56 | sean-k-mooney | it look like ubuntu just enables it by defualt to be safe by defualt | |
| 17:31:23 | kashyap | sean-k-mooney: RHEL doesn't either, BTW. And one can configure what action 'libvirt-guests' can take on host shutdown | |
| 17:31:45 | sean-k-mooney | ok | |
| 17:32:01 | sean-k-mooney | well that is not related to the issue mriedem was having | |
| 17:32:17 | sean-k-mooney | the issue he was having was that the vm hung | |
| 17:32:49 | sean-k-mooney | and then the linux-guests scipt also hugn on shutdown for the same reason | |
| 17:33:09 | sean-k-mooney | ther eis a timout in the service file if i rememebr correctly it waits for up to 2 minutes | |
| 17:33:28 | sean-k-mooney | and it continue with the system shutdown if it takes longer then that | |
| 17:34:06 | kashyap | I wasn't saying it is related. On your FS corruption: no, it is admin / higher-level tool's responsibility to ensure your guests will quiesce its FS. | |
| 17:35:16 | kashyap | (And yes, there is a timeout: check SHUTDOWN_TIMEOUT in /etc/sysconfig/libvirt-guests) | |
| 17:35:17 | sean-k-mooney | ack | |
| 17:35:24 | kashyap | Default is 5 minutes. | |
| 17:35:56 | sean-k-mooney | ya i have seen it when i have rebooted system in the console output in the past i just noticed it had one but never really look that closely | |
| 17:39:23 | melwitt | dansmith, mriedem: would like to have your review on a change to remove the "last context manager" from the CellDatabases fixture https://review.opendev.org/672604. this came up again while I was working on adding a func test to Kevin_Zheng's multi-cell nova-manage db archive_deleted_rows patch https://review.opendev.org/507486, which has been of high priority interest downstream lately | |
| 17:39:30 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Process [compute] in $NOVA_CPU_CONF in nova-next https://review.opendev.org/672800 | |
| 17:42:42 | sean-k-mooney | melwitt: by the way has anyone reviewed the unified limits spec for the api subteam? not sure who that would be | |
| 17:44:34 | efried | sean-k-mooney: Isn't gmann "the api subteam"? | |
| 17:45:42 | sean-k-mooney | efried: i guess so? i wasnt sure who was on it. but i didnt want too see that sepc slip through the cracks if they were not about to review it | |
| 17:46:01 | efried | Agree. | |
| 17:46:15 | melwitt | sean-k-mooney: no, not yet. people I usually ask about api stuff are alex_xu, gmann | |
| 17:46:16 | sean-k-mooney | similarly with the image encryption spec. | |
| 17:46:16 | efried | I think I added gmann to that spec for that reason, but not sure if he looked. | |
| 17:46:57 | efried | I'm sort of delegating, like "encouraging" the spec owners to track down whoever is needed. | |
| 17:48:09 | sean-k-mooney | oh the provider yaml spec merged | |
| 17:48:31 | sean-k-mooney | cool i should read the final version | |
| 17:49:51 | sean-k-mooney | johnthetubaguy: melwitt: if ye feel like reviewing a spec that is close https://review.opendev.org/#/c/608696/ im happy to do a little reaching out on behalf of the spec owner :) | |
| 18:14:31 | dansmith | melwitt: okay, I'm generally pretty wary of changing that stuff (or even trying to load enough context to review that). I'm not sure I'll get to that point before I Ieave tomorrow, but...ack :) | |
| 18:16:40 | openstackgerrit | Merged openstack/nova master: Remove 'nova.virt.driver.ComputeDriver.estimate_instance_overhead' https://review.opendev.org/672106 | |
| 18:18:34 | melwitt | dansmith: ok, thanks for letting me know. it was tough for me to load the context myself, so I understand. I wanted to ideally have you review since the patch involves ripping out the stuff that you had to add with the _cell_lock | |
| 18:20:26 | melwitt | I think it makes the fixture much simpler, but definitely want to run it by you in case I missed something | |
| 18:22:13 | dansmith | yeah I'm just afraid of it breaking something subtle which we don't find for a couple months and then think we need to fix it by changing the real code when in fact the fixture is too relaxed or something | |