| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-25 | |||
| 17:07:38 | mriedem | the domain | |
| 17:07:47 | mriedem | $ sudo virsh list --all | |
| 17:07:47 | mriedem | Id Name State | |
| 17:07:47 | mriedem | ---------------------------------------------------- | |
| 17:07:47 | mriedem | 3 instance-00000003 paused | |
| 17:08:03 | sean-k-mooney | and in nova its active | |
| 17:08:09 | mriedem | no | |
| 17:08:20 | mriedem | it's building b/c the libvirt driver is waiting for the power state to change from paused to running | |
| 17:09:30 | sean-k-mooney | oh and you it alwready told it to unpause? we start the domian in the pased state. i wonder if the qemu monitor has hung | |
| 17:14:14 | mriedem | fun | |
| 17:14:14 | mriedem | Jul 25 17:13:36 devstack libvirt-guests.sh[18879]: Timeout expired while shutting down domains | |
| 17:14:14 | mriedem | Jul 25 17:13:36 devstack systemd[1]: libvirt-guests.service: Control process exited, code=exited status=1 | |
| 17:14:21 | mriedem | trying to restart libvirt-guests | |
| 17:15:19 | sean-k-mooney | is this a clean install of ubuntu 18.04? | |
| 17:15:37 | mriedem | well from a vexxhost image of 18.04 but yeah | |
| 17:16:15 | sean-k-mooney | strange i personlly havent had any issue with 18.04 i did an install fiday | |
| 17:16:33 | mriedem | me neither | |
| 17:17:57 | sean-k-mooney | are the vexhost image available for download | |
| 17:18:06 | mriedem | idk, i'm trashing this vm | |
| 17:19:12 | mnaser | the vexxhost images are straight up the ones shipped by ubuntu | |
| 17:19:12 | sean-k-mooney | ya i would just start over too to be honest. i suspect its somehitng to doe with libvirt/qemu or maybe apparmor but i would start clean | |
| 17:19:27 | sean-k-mooney | mnaser: the cloud images | |
| 17:19:32 | mnaser | yep | |
| 17:19:38 | mnaser | only thign we do is convert from qemu to raw | |
| 17:19:41 | mnaser | and upload | |
| 17:19:53 | sean-k-mooney | im guessing ye are using ceph as a backend then | |
| 17:20:12 | mnaser | indeed :) | |
| 17:20:13 | kashyap | On Fedora, the 'libvirt-guests' thing isn't even enabled: | |
| 17:20:15 | kashyap | $> systemctl status libvirt-guests | |
| 17:20:15 | kashyap | ● libvirt-guests.service - Suspend/Resume Running libvirt Guests | |
| 17:20:15 | kashyap | Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; disabled; vendor preset: disabled) | |
| 17:20:18 | kashyap | Active: inactive (dead) | |
| 17:20:21 | kashyap | ... | |
| 17:20:46 | kashyap | But yeah, that timeout of 'libvirt-guests' looks spurious enough, might as well start over. | |
| 17:21:05 | sean-k-mooney | kashyap: would that not cause filesystem curruption if you did not suspend them on rebooting the host | |
| 17:21:06 | kashyap | (Also, not sure if that paused instance's QEMU process went 'defunct') | |
| 17:21:37 | efried | following up re os-vif and python-novaclient releases: Libs are required to do one release per milestone. os-vif was last released at m1, so we can expect the release team to propose that one. python-novaclient was released a couple weeks ago, so we're probably good on that one. | |
| 17:21:41 | sean-k-mooney | kashyap: im guessing the qemu moniotr process stoped processing messages form libvirt | |
| 17:22:47 | sean-k-mooney | efried: ok there is one think i would like to fix soonish but im only starting on it today | |
| 17:22:48 | kashyap | Yeah, but that doesn't tell us why. It could be any no. of reasons | |
| 17:23:33 | sean-k-mooney | kashyap: yep its proably quicker to kill it and spin up a clean vm | |
| 17:23:45 | sean-k-mooney | if mriedem hits it again we can take another look | |
| 17:24:22 | kashyap | sean-k-mooney: Yeah, on FS corruption, possibly "enterprise distros" would enable it | |
| 17:24:56 | sean-k-mooney | it look like ubuntu just enables it by defualt to be safe by defualt | |
| 17:31:23 | kashyap | sean-k-mooney: RHEL doesn't either, BTW. And one can configure what action 'libvirt-guests' can take on host shutdown | |
| 17:31:45 | sean-k-mooney | ok | |
| 17:32:01 | sean-k-mooney | well that is not related to the issue mriedem was having | |
| 17:32:17 | sean-k-mooney | the issue he was having was that the vm hung | |
| 17:32:49 | sean-k-mooney | and then the linux-guests scipt also hugn on shutdown for the same reason | |
| 17:33:09 | sean-k-mooney | ther eis a timout in the service file if i rememebr correctly it waits for up to 2 minutes | |
| 17:33:28 | sean-k-mooney | and it continue with the system shutdown if it takes longer then that | |
| 17:34:06 | kashyap | I wasn't saying it is related. On your FS corruption: no, it is admin / higher-level tool's responsibility to ensure your guests will quiesce its FS. | |
| 17:35:16 | kashyap | (And yes, there is a timeout: check SHUTDOWN_TIMEOUT in /etc/sysconfig/libvirt-guests) | |
| 17:35:17 | sean-k-mooney | ack | |
| 17:35:24 | kashyap | Default is 5 minutes. | |
| 17:35:56 | sean-k-mooney | ya i have seen it when i have rebooted system in the console output in the past i just noticed it had one but never really look that closely | |
| 17:39:23 | melwitt | dansmith, mriedem: would like to have your review on a change to remove the "last context manager" from the CellDatabases fixture https://review.opendev.org/672604. this came up again while I was working on adding a func test to Kevin_Zheng's multi-cell nova-manage db archive_deleted_rows patch https://review.opendev.org/507486, which has been of high priority interest downstream lately | |
| 17:39:30 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Process [compute] in $NOVA_CPU_CONF in nova-next https://review.opendev.org/672800 | |
| 17:42:42 | sean-k-mooney | melwitt: by the way has anyone reviewed the unified limits spec for the api subteam? not sure who that would be | |
| 17:44:34 | efried | sean-k-mooney: Isn't gmann "the api subteam"? | |
| 17:45:42 | sean-k-mooney | efried: i guess so? i wasnt sure who was on it. but i didnt want too see that sepc slip through the cracks if they were not about to review it | |
| 17:46:01 | efried | Agree. | |
| 17:46:15 | melwitt | sean-k-mooney: no, not yet. people I usually ask about api stuff are alex_xu, gmann | |
| 17:46:16 | efried | I think I added gmann to that spec for that reason, but not sure if he looked. | |
| 17:46:16 | sean-k-mooney | similarly with the image encryption spec. | |
| 17:46:57 | efried | I'm sort of delegating, like "encouraging" the spec owners to track down whoever is needed. | |
| 17:48:09 | sean-k-mooney | oh the provider yaml spec merged | |
| 17:48:31 | sean-k-mooney | cool i should read the final version | |
| 17:49:51 | sean-k-mooney | johnthetubaguy: melwitt: if ye feel like reviewing a spec that is close https://review.opendev.org/#/c/608696/ im happy to do a little reaching out on behalf of the spec owner :) | |
| 18:14:31 | dansmith | melwitt: okay, I'm generally pretty wary of changing that stuff (or even trying to load enough context to review that). I'm not sure I'll get to that point before I Ieave tomorrow, but...ack :) | |
| 18:16:40 | openstackgerrit | Merged openstack/nova master: Remove 'nova.virt.driver.ComputeDriver.estimate_instance_overhead' https://review.opendev.org/672106 | |
| 18:18:34 | melwitt | dansmith: ok, thanks for letting me know. it was tough for me to load the context myself, so I understand. I wanted to ideally have you review since the patch involves ripping out the stuff that you had to add with the _cell_lock | |
| 18:20:26 | melwitt | I think it makes the fixture much simpler, but definitely want to run it by you in case I missed something | |
| 18:22:13 | dansmith | yeah I'm just afraid of it breaking something subtle which we don't find for a couple months and then think we need to fix it by changing the real code when in fact the fixture is too relaxed or something | |
| 18:22:26 | dansmith | but that's just because of how hard it was to get it right in the first place, of course | |
| 18:27:00 | melwitt | oh, you mean something to do with racing tests appearing like real bugs when there's really just an issue with the fixture? yeah, I can understand that concern. as far as I can tell, my proposed patch removes all changing of global state, so I'd think there won't be an issue. but those are famous last words, I know | |
| 18:29:44 | melwitt | if we're too afraid to change the fixture, then we will hopefully be able to accept the multi-cell nova-manage patch's func test not being full coverage because of the faking that the CellDatabases fixture does. my primary objective is to get the multi-cell nova-manage db archive_deleted_rows done | |
| 18:31:33 | dansmith | okay, I'm not sure why that is harder than other cell iteration things we do in tests, but I'd be much more inclined to accept more mockery (since that's really a trivial operation) vs. blocking that on rearchitecting the fixture. But, I haven't looked enough into why that's a problem to say really | |
| 18:31:33 | melwitt | and while working on that, its func test was not failing when it should have been (bug in a patchset), and I found it wasn't failing properly because of the "last context manager" faking in the fixture | |
| 18:31:40 | dansmith | maybe I can try to do that before tomorrow at least | |
| 18:34:30 | melwitt | dansmith: tl;dr is the func test is written correctly and is good, but it did _not_ catch a bug in the proposed multi-cell archive impl because of the faking in the fixture. the fixture auto-targets untargeted database access to the last targeted database or the default database. the former hid the bug in the impl because the fixture auto-targeted something that was not targeted in real life and needed to be targeted in real life | |
| 18:34:57 | melwitt | I hope that makes sense | |
| 18:35:02 | mriedem | i'd also rather figure out why the func tests on the archive patch don't work rather than block on redoing the fixture, but i don't know what the issues were, | |
| 18:35:04 | mriedem | having said that, | |
| 18:35:16 | mriedem | i have a func test in my cross-cell resize series that does db archive on all 3 cells in the test https://review.opendev.org/#/c/651650/22/nova/tests/functional/test_cross_cell_migrate.py | |
| 18:35:20 | dansmith | but the real code sends untargeted stuff to the default db in the config, which is why the fixture does | |
| 18:35:36 | melwitt | mriedem: I did figure it out, it was because of the auto-targeting by "last targeted database" if untargeted | |
| 18:36:04 | dansmith | mriedem: yeah, even still, I'd be happy with just a unit test to make sure that archive is calling archive on all the cell mappings.. it | |
| 18:36:09 | dansmith | is such a trivial op I don't really know that we need much more than that, | |
| 18:36:17 | mriedem | ok i guess i mean "i'd rather figure out an easier way to make the new tests work with the existing fixture" | |
| 18:36:24 | melwitt | dansmith: but the fixture also sends untargeted stuff to the "last targeted db" first. I think it should only send untargeted stuff to the default db | |
| 18:36:24 | dansmith | and we run archive in tempest jobs, which should hit cell0 and cell1 if we make it run all cells | |
| 18:36:25 | mriedem | also, | |
| 18:36:29 | melwitt | (which is what my patch is doing) | |
| 18:36:48 | mriedem | i was going to say - nova-next runs archive_deleted_rows, we can and should make it run on all cells | |
| 18:36:54 | mriedem | which will hit cell0 and cell1 as dansmith said | |
| 18:37:03 | mriedem | so....then we'd have real integration test coverage | |
| 18:37:04 | dansmith | melwitt: everything in the compute node is untargeted though, which is why I don't really know how you can change that globally and have it match the real world | |
| 18:37:09 | dansmith | but.. I haven't read it so I dunno | |
| 18:37:11 | mriedem | of the cli, which is better than the functional stuff anyway | |