| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-09 | |||
| 14:39:30 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: default to max version when no session https://review.opendev.org/c/openstack/osc-placement/+/794276 | |
| 14:40:11 | gibi | sean-k-mooney: thanks. just added the task number to the commit message too ^^ | |
| 14:40:25 | gibi | this will connect the review to the task in storyboard | |
| 14:40:28 | sean-k-mooney | oh ok i have seen both but i tought task was optional | |
| 14:40:30 | gibi | and update its state | |
| 14:40:33 | sean-k-mooney | oh ok | |
| 14:40:37 | sean-k-mooney | cool | |
| 14:40:56 | sean-k-mooney | am general question | |
| 14:41:13 | sean-k-mooney | if placment is going to remain a deliverable of compute | |
| 14:41:31 | sean-k-mooney | do we want to continue to maintain both storyboard and lauchpad | |
| 14:41:54 | sean-k-mooney | i also just realised the placement repariting spec should have been proposed in the placement repo | |
| 14:42:06 | sean-k-mooney | since that is where placment specs are ment to be tracked | |
| 14:42:35 | sean-k-mooney | https://docs.openstack.org/placement/latest/specs/index.html | |
| 14:42:35 | gibi | sean-k-mooney: I checked the placement repo but it does not have specs folder | |
| 14:42:51 | sean-k-mooney | its in the docs folder | |
| 14:42:55 | sean-k-mooney | so that it renders there | |
| 14:42:56 | gibi | /o\ | |
| 14:43:12 | sean-k-mooney | https://github.com/openstack/placement/tree/master/doc/source/specs | |
| 14:43:49 | sean-k-mooney | i kind of forgot this was there until i rememebred a few days ago | |
| 14:44:52 | gibi | OK, I will move the re-prenting spec to the placement repo :/ | |
| 14:45:04 | sean-k-mooney | well i was not going to ask you to do that | |
| 14:45:18 | sean-k-mooney | just going forward do we want to keep the split | |
| 14:45:39 | sean-k-mooney | e.g. nova has nova-specs and launchpad placement has in repo specs and sotry borad | |
| 14:46:03 | sean-k-mooney | this is more a wider team question or mailing list topic | |
| 14:46:17 | gibi | I don't feel the power to do a migration from storyboard | |
| 14:46:21 | sean-k-mooney | but that bug just reminded me of it | |
| 14:46:59 | gibi | we have very few incoming things in placemnet | |
| 14:47:42 | sean-k-mooney | true | |
| 14:47:55 | sean-k-mooney | do we want to add it to the meeting adgenda | |
| 14:48:08 | sean-k-mooney | we have the bug review for nova in lanuch pad | |
| 14:48:17 | sean-k-mooney | do we have somethign that will do the same for story borad | |
| 14:48:37 | sean-k-mooney | e.g. a linke to the open stories | |
| 14:48:58 | sean-k-mooney | its pretty stable and well tested so placment bugs are rare | |
| 14:50:55 | gibi | sean-k-mooney: please put it up to the agenda. Honestly I don't think it worth the effort to do extra things with placement but it make sense to discuss it | |
| 14:52:12 | sean-k-mooney | oh i was suggesting adding a "check for open placment bugs" item | |
| 14:52:19 | sean-k-mooney | like when we do the nova bug review | |
| 14:52:38 | sean-k-mooney | more just to prompt use to look for placment bugs if tehy areever reported | |
| 14:53:00 | sean-k-mooney | most get brought up here anyway since nova typicly is what finds them if they exist | |
| 14:57:13 | gibi | honestly I don't belive that people will start looking at open bug reports in placement by that prompt. but I might be negative | |
| 15:01:31 | sean-k-mooney | ok then lets leave it be so | |
| 15:03:43 | melwitt | tosky: guh, ok, I was wondering if there might be something like that since the fails. I'll look into a potential squash, thanks for the hint | |
| 15:04:00 | gibi | I just don't have the bandwidth to do a proper care for placement as well, but it seem that placement does not require that level of care most of the time | |
| 15:05:02 | masterpe[m] | Hi, we have Train running, yesterday we got the error "No valid host was found. There are not enough hosts available" on our platform. If I count the free RAM and vcpu of the nodes, I should have enough resources left to get the instances started. Is there something that I miss? | |
| 15:05:36 | melwitt | sean-k-mooney, gibi: I knew that placement had its own specs repo but I thought it was intentional that proposed spec to the nova-specs repo since the projects are back together again /shrug | |
| 15:09:53 | melwitt | gibi: there is a review I have hanging around if you might be able to spare some time, it is not urgent https://review.opendev.org/c/openstack/nova/+/780395 | |
| 15:15:38 | sean-k-mooney | melwitt: ya so i didnt know i we had decied to not use the in repos specs or not | |
| 15:15:54 | sean-k-mooney | e.g. if we had jsut defaulted to nova-spec or if it was intentional | |
| 15:16:31 | melwitt | sean-k-mooney: no, I'm not saying there was a decision, I had assumed something like that had occurred because of where the re-parent spec was proposed, but I was wrong | |
| 15:19:52 | gibi | melwitt: queued that up for review | |
| 15:20:24 | melwitt | thank you gibi | |
| 15:26:23 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575 | |
| 15:26:24 | melwitt | masterpe[m]: you need to look at the debug level logs in the placement api logs and the nova-scheduler logs to see details about why no valid host was found | |
| 15:28:39 | masterpe[m] | To solve it for now I enabled a node that was disabled/marked as spare. So I need to enable debug mode to troubleshoot? | |
| 15:32:42 | melwitt | masterpe[m]: to get full detail, yes. what you can do without debug mode is check whether there is a message like "no allocation candidates" in the nova-scheduler log, if you see that, it's likely cpu/ram/disk (as those resources are filtered by placement service) | |
| 15:33:42 | melwitt | if you don't see that, it means one or more of the nova side scheduler filters eliminated all hosts | |
| 15:34:11 | opendevreview | Balazs Gibizer proposed openstack/placement master: [DNM]: test consumer_type rollback due to consumer_gen conflict https://review.opendev.org/c/openstack/placement/+/795577 | |
| 15:36:31 | melwitt | gibi: oh heh, I had wondered if you saw my latest confusing comment. I was thinking the same thing this morning that it would be best to check what happens with a test. thanks for doing that | |
| 15:36:59 | gibi | melwitt: yeah, you are right the whole PUT /allocations/{uuid} call is not a single transaction | |
| 15:37:28 | gibi | so with the current implementation we can leak a consumer_type update from a rejected call :/ | |
| 15:39:31 | gibi | unfortunatly create a test case for a real race is too hard | |
| 15:39:43 | melwitt | gibi: ah, I see your new comment on the review. yeah ok, agree we need something higher up the stack then in order to not leak (or some other better idea yet to be thought of) :( I will work on it | |
| 15:40:32 | gibi | it is hard :/ | |
| 15:42:25 | melwitt | yeah, would require faking things with mocks at best. I didn't explore that yet so not sure how messy or possible | |
| 15:46:49 | spatel | sean-k-mooney periodically i am seeing this issue on my nova logs not sure if this is bug or not - http://paste.openstack.org/show/806496/ | |
| 15:51:45 | melwitt | spatel: that error means connection to the database failed while trying to write a service record update (heartbeat), not a bug. error being emitted from pymysql client | |
| 15:52:41 | spatel | melwitt today i said terraform to build 20 vm and 4 vm failed and stick in BUILD process, when i check their compute nodes i found that error | |
| 15:55:44 | melwitt | spatel: that paste wouldn't be directly related to a build failure (that paste is a service record update for a service heartbeat) so you'd need to look at which errors are related to the build fail, if any | |
| 16:03:13 | spatel | let me see, may be issue of rabbitmq in that case | |
| 16:05:41 | melwitt | dansmith: someone linked this review in the freenode channel and I thought you might have opinion about it. it's a proposal to allow config of rbd direct snapshot to enable an operator to disallow the fast clone, with the use case being the operator is migrating the backend and the fast clone wouldn't honor the glance backend config (I don't fully understand this part) https://review.opendev.org/c/openstack/nova/+/794837 | |
| 16:30:53 | spatel | melwitt also noticed periodically rabbitMQ connection reset error from nova-api - http://paste.openstack.org/show/806498/ | |
| 16:31:06 | spatel | is this a normal behavior ? | |
| 16:35:44 | melwitt | spatel: it's not uncommon but it's not desirable. I have seen it many times, related to environmental issues like network disruptions. it's also similar to the DBConnectionError you pasted earlier, that one was about the pymysql client failing to be able to connect to the database, these rabbit errors are showing connection reset or closed trying to connect to rabbitmq | |
| 16:37:05 | spatel | These controller and compute node on same rack | |
| 16:37:19 | spatel | let me restart my whole rabbitMQ cluster | |
| 16:38:32 | spatel | when i delete vm they getting stuck in deleting state | |
| 17:14:36 | dansmith | melwitt: ack, thanks | |
| 17:39:55 | opendevreview | Artom Lifshitz proposed openstack/nova master: docs: Explicitly state lack of support for node renaming https://review.opendev.org/c/openstack/nova/+/795607 | |
| 17:58:43 | artom | Huh, looks like the docs gate is broken on nova/crypto.py | |
| 17:58:48 | artom | ^^ somehow failed it | |
| 18:03:10 | artom | Actually, just pep8 in general is broken: https://zuul.opendev.org/t/openstack/build/0b3b9315c9cb4d489f2ffb1355145a99 | |
| 18:04:32 | artom | Ah, it's mypy... | |
| 18:16:14 | lyarwood | artom: the fix for that is in the gate | |
| 18:16:44 | artom | lyarwood, oh. And here I was filing a bug and writing a fix. | |
| 18:16:44 | lyarwood | https://review.opendev.org/c/openstack/nova/+/795533 | |
| 18:17:41 | lyarwood | Always enjoyable when that happens ^_^ | |
| 18:18:23 | artom | I mean, no mailing list message, no bug filed, nothing :P | |
| 18:18:57 | artom | Hrmm, I wonder if my fix could stiff have value | |
| 18:21:26 | artom | I guess not, since we'll need to manually update requirements anyways? | |
| 18:21:54 | artom | I did it by just adding `python -m mypy --install-types` to tools/mypywrap.sh | |
| 18:23:31 | lyarwood | Yeah I think stephenfin tried that earlier | |
| 18:23:40 | lyarwood | but it wasn't enough in the gate for some reason | |
| 18:24:12 | artom | I guess because lower-constraints.txt? | |
| 18:42:59 | cz3 | This is probably a trivial question... I wanted to import Nova modules into interactive Python shell, but I cannot figure out how to load the config from default nova.conf. Is `CONF = nova.conf.CONF` supposed to handle this? | |
| 18:43:54 | cz3 | I wanted to play around with objects module through interactive shell but the lack of loaded config is blocking me | |
| 18:45:45 | artom | cz3, never tried it myself, but looking at the code, yeah, import nova.conf; CONF = nova.conf.CONF should do it | |
| 18:49:26 | cz3 | yeah, that's what I tried... doesn't seem to work. looks like it is still trying to use the defaults. I assume that because it tries to look up things in sqlite instead of mysql that I have in nova.conf | |
| 18:51:00 | artom | Maybe it's not finding the config file, and using the defaults... | |
| 18:52:30 | artom | In fact, if you're importing it via Python (and not via running a Python executable on the command line), I'm not even sure how it finds the config file to use... | |
| 18:54:21 | artom | I *think* you want to do CONF = nova.conf.CONF(project='nova') in such a case... | |
| 18:54:29 | artom | And then it'll look for nova.conf in the "normal" places | |