| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-17 | |||
| 13:38:26 | sean-k-mooney | not that i expect nova specs to rendered to pdf like some of the other docs but its nice to be consitent | |
| 13:38:40 | ralonsoh | sean-k-mooney, you mean https://bit.ly/2XN4D7f ?? | |
| 13:39:15 | sean-k-mooney | ya i think that is nicer to read and eisier to serach for if you are reading the rst file directly | |
| 13:39:20 | ralonsoh | okidoki | |
| 13:39:41 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: func test for migrate server with ports having resource request https://review.opendev.org/655113 | |
| 13:40:20 | sean-k-mooney | i almost exclisively read teh docs and sepc in gerrit/locally without rendering so i have a bias to that form. | |
| 13:42:54 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Note use of 'nova-manage db sync --config-file' https://review.opendev.org/671298 | |
| 13:43:21 | stephenfin | sean-k-mooney: You'll probably like that ^ | |
| 13:43:27 | stephenfin | Also | |
| 13:44:13 | stephenfin | I forget just how good we've got it in OpenStack whenever I look at community projects | |
| 13:44:33 | stephenfin | "Why are half your tests skipped?" "Oh, they just stopped working so we disabled them" | |
| 13:44:34 | stephenfin | :) | |
| 13:45:29 | cdent | sweet | |
| 13:47:59 | sean-k-mooney | stephenfin: +1'd what had you running that manually anwyway? just trying to avoid restacking? | |
| 13:48:39 | stephenfin | I'm still working on the cpu-resources spec. I'd stacked with master and then wanted to work through an upgrade | |
| 13:48:52 | stephenfin | I foresee a lot of restacks in the coming week or so | |
| 13:49:24 | stephenfin | But! http://paste.openstack.org/show/754499/ | |
| 13:50:26 | Nick_A | Hello - we have instance images and snapshots set to qcow2 in nova.conf. Is there a way to bypass what appears to be an unnecessary conversion step before it uploads the snapshot to glance? Am I right in understanding that it's trying to convert qcow2 > qcow2? | |
| 14:09:36 | openstackgerrit | Artom Lifshitz proposed openstack/nova stable/rocky: [DNM] testing bug/1813789 revert resize events https://review.opendev.org/671303 | |
| 14:12:49 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova-specs master: Boot a VM with an unaddressed port https://review.opendev.org/641670 | |
| 14:45:27 | Zara | hullo! I'm back with my '`nova list` slow when run with member user but not admin user' question-- narrowed it down a bit; the slow requests are the ones via the neutronclient to ports.json (as part of nova's servers/detail request). 10s apiece for member; 200ms apiece for admin. maybe a neutron or keystone issue? | |
| 14:46:45 | sean-k-mooney | Zara: did you check if the delay was in the neuron api or in nova | |
| 14:59:42 | Zara | ahhhh, listing ports alone is also slow, so almost certainly neutron (so I'm even more in-the-wrong-channel than I thought; sorry!) | |
| 15:00:04 | sean-k-mooney | haha no worries | |
| 15:01:09 | Zara | :) | |
| 15:28:46 | alex_xu | dansmith: I faced a problem. Seems the dst host's libvirt won't keep the dst domain xml when live-migration failed due to the dst host down. So when the dst host's nova-compute startup again, it can't know the target vpmem which may dirty by the live-migration, then can't do a cleanup for that vpmem. | |
| 15:28:51 | alex_xu | efried: ^ | |
| 15:29:25 | dansmith | um | |
| 15:29:38 | dansmith | you mean live migration, which fails in the middle because the compute host goes down or something? | |
| 15:29:44 | efried | yes | |
| 15:29:52 | alex_xu | yes, the dst compute host goes down | |
| 15:30:36 | sean-k-mooney | well that should be reverted right. so i guess the problem is some of the data could have been copied | |
| 15:30:51 | alex_xu | yes | |
| 15:31:08 | sean-k-mooney | alex_xu: could we optionally clean the unsed namespaces on agent start | |
| 15:31:09 | dansmith | yeah, I mean, that's a serious failure, and all kinds of things could go wrong there | |
| 15:31:23 | dansmith | sean-k-mooney: yeah was just going to say.. that's probably safest | |
| 15:31:39 | alex_xu | ok, but it should be slow | |
| 15:31:51 | sean-k-mooney | we could proably store a file somewhere of things we are currently programing and delete the file if we completed properly | |
| 15:32:03 | alex_xu | it is ok to cleanup, since the live-migration won't be continue again | |
| 15:32:10 | dansmith | because even if you were tracking them some other way, that failure scenario, followed by a db archive before the compute node starts up again would result in not knowing about the previous failure | |
| 15:32:34 | dansmith | sean-k-mooney: yeah, there may be residue just from the incoming disk images and stuff actually | |
| 15:32:38 | dansmith | also, | |
| 15:32:44 | dansmith | there should be an uncompleted migration record in the db | |
| 15:32:52 | dansmith | which we already look for in a couple situations to clean up | |
| 15:33:17 | sean-k-mooney | the migration record wont have the vpmem info unfortuately | |
| 15:33:20 | dansmith | so maybe if there are aborted migrations against us when we start, we do a full clean for safety or something? | |
| 15:33:24 | dansmith | no, I know | |
| 15:33:39 | sean-k-mooney | so we really need to write somethign to the compute node disk to keep track of it | |
| 15:34:38 | sean-k-mooney | e.g. we need to store it locally incase teh instance is delete and archive before we restart teh agaent on the failed node | |
| 15:34:41 | dansmith | ...or just clean all unused namespaces on startup if we see an aborted live migration | |
| 15:34:47 | efried | ++ | |
| 15:34:51 | sean-k-mooney | ya that too | |
| 15:34:56 | efried | alex_xu: ^ | |
| 15:35:11 | alex_xu | yea, I'm ok with that | |
| 15:35:20 | dansmith | why is it slow btw? | |
| 15:35:22 | sean-k-mooney | but alex_xu was concerned about the time. i guess if its done asyc outside of the agent then its fine | |
| 15:35:23 | efried | need some way to mark them unavailable for spawns | |
| 15:35:30 | dansmith | shouldn't this be a TRIM type operation on some silicon? | |
| 15:35:55 | efried | don't you have to write zeroes to the whole thing, or something? | |
| 15:35:57 | dansmith | efried: block compute startup until it's finished like all our other "recover from failure" type things | |
| 15:36:03 | efried | ah, nice | |
| 15:36:04 | dansmith | efried: shouldn't, that would be terrible | |
| 15:36:06 | alex_xu | no..just write zero to each bit... | |
| 15:36:15 | dansmith | that's ridiculous | |
| 15:36:29 | dansmith | isn't this effectively flash? | |
| 15:37:14 | sean-k-mooney | dansmith: well we would have to do a secure erase rather then trim but ya optane is like flash but different | |
| 15:37:35 | alex_xu | I guess there is no cache, since it is write by the DAX way | |
| 15:37:54 | dansmith | sean-k-mooney: either, trim or dump the key.. either should be fast and avoid wearing the flash to friggin write zeroes | |
| 15:38:12 | dansmith | sean-k-mooney: so does that mean every time we destroy an instance using one of these that we have to block destroy until we've written gigabytes to this device? | |
| 15:38:20 | sean-k-mooney | alex_xu: i assume we can delgate the erase to a cli too and just do it async or in parallel | |
| 15:38:33 | dansmith | whereas if we were giving them flash we could just run a single trim command and move on? | |
| 15:38:35 | alex_xu | we are using the daxio | |
| 15:38:49 | sean-k-mooney | dansmith: i mean technically yes | |
| 15:39:11 | sean-k-mooney | that would be similar to the lvm image backend erase | |
| 15:39:27 | dansmith | sean-k-mooney: right, which is ridiculous and based on 1980s notions of disk, so I give it a pass | |
| 15:39:58 | dansmith | and also not really enjoyed by operators everywhere :) | |
| 15:40:10 | sean-k-mooney | alex_xu: to dansmith's point if we are using daxio can we just tell the device to nuke everythin in the namespace | |
| 15:40:33 | alex_xu | no, I don't think we have that kind of TRIM | |
| 15:40:43 | efried | this seems like a separate issue though. Whatever "clear" we have to do, we have to do the same thing on destroy() | |
| 15:40:55 | sean-k-mooney | ya | |
| 15:40:59 | sean-k-mooney | that is fair | |
| 15:41:07 | dansmith | efried: sure, it's more just that if we're going to massively slow down destroy, it'd be good to know that | |
| 15:41:15 | dansmith | so we can mock this "feature" if nothing else | |
| 15:41:22 | sean-k-mooney | the question that raises is can we have the agent wait to start up or not | |
| 15:41:23 | efried | IIRC that was in the spec | |
| 15:41:23 | dansmith | jay would be rolling over in his grave | |
| 15:41:38 | sean-k-mooney | could we play with the reserved value in placment | |
| 15:41:40 | dansmith | sean-k-mooney: we can and should, like anything else we do on this | |
| 15:41:50 | dansmith | I'm less concerned about the delayed compute startup than I am with destroy | |
| 15:42:11 | sean-k-mooney | e.g. make the dirty namespces reserved until we clean it | |
| 15:42:26 | dansmith | if you're billing by the hour, and a bunch of people destroy instances all at once and they all have to wait in line to have zeroes written to this new whizbang memory thing... | |
| 15:42:53 | efried | sean-k-mooney: if we change inventory on destroy() we're going to run into cdent's generation conflict race all over the place. | |
| 15:42:54 | sean-k-mooney | for what its worth it can do that at several GB/S | |
| 15:42:54 | dansmith | sean-k-mooney: that's a lot more complex and more opportunity to be interrupted in the middle and needing to be properly resumed | |
| 15:43:23 | sean-k-mooney | ok then lets just keep it simple | |
| 15:43:43 | cdent | (my patch to not do that, controlled by config, got merged internally) | |
| 15:43:44 | sean-k-mooney | and block untill they are all cleaned if their was a faild migration as ye suggested | |
| 15:44:03 | dansmith | sean-k-mooney: ++ | |
| 15:45:05 | sean-k-mooney | apparently the frist gen stuff will hit 8.9 GB/s in sequtial writes | |
| 15:45:29 | sean-k-mooney | although the cleaning operation would likely impact the performce of other isntances if you tried to max it out | |