| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-09 | |||
| 13:46:13 | opendevreview | Stephen Finucane proposed openstack/nova master: requirements: Add types-paramiko https://review.opendev.org/c/openstack/nova/+/795533 | |
| 13:46:14 | opendevreview | Stephen Finucane proposed openstack/nova master: tox: Install mypy type stubs https://review.opendev.org/c/openstack/nova/+/795544 | |
| 13:46:24 | stephenfin | lyarwood, gibi: Alternate approach ^ | |
| 14:12:05 | spatel | sean-k-mooney morning! i am trying to delete one of vm which doesn't existing on compute machine (i can't see it on compute host). i am not able to delete it, any good way to clean up ? | |
| 14:13:14 | gibi | stephenfin: https://review.opendev.org/c/openstack/nova/+/795544 doesnt seem to work | |
| 14:17:41 | stephenfin | oh, well then | |
| 14:17:58 | stephenfin | so I guess you need to run it, let it build a cache, then run it again to install the missing deps | |
| 14:18:04 | stephenfin | so that won't work | |
| 14:18:10 | stephenfin | pity :-( | |
| 14:19:19 | opendevreview | Stephen Finucane proposed openstack/nova master: requirements: Add types-paramiko https://review.opendev.org/c/openstack/nova/+/795533 | |
| 14:19:46 | stephenfin | back to v2 we go | |
| 14:24:35 | sean-k-mooney | spatel: i think we have force delete option | |
| 14:24:56 | spatel | hmm --force ? | |
| 14:25:04 | sean-k-mooney | spatel: if the vm no longere exits on the compute node but is still in the nova db that is what i woudl try next | |
| 14:25:21 | spatel | perfect! let me try | |
| 14:25:40 | sean-k-mooney | well i was thinking of "nova force-delete <server uuid>" | |
| 14:25:45 | sean-k-mooney | let me check if its in osc | |
| 14:26:06 | sean-k-mooney | yes there is a --force | |
| 14:26:19 | sean-k-mooney | os openstack server delete --force <uuid> | |
| 14:27:00 | spatel | penstack server delete: error: unrecognized arguments: --force | |
| 14:27:12 | spatel | I am running victoria | |
| 14:28:27 | spatel | nova force-delete works! | |
| 14:29:44 | spatel | sean-k-mooney openstack server delete doesn't have --force option | |
| 14:30:00 | sean-k-mooney | it does in the latest release | |
| 14:30:34 | sean-k-mooney | spatel: stephenfin added it along with closing most of the other gaps recently | |
| 14:31:24 | spatel | may be available in current release but not in victoria | |
| 14:31:56 | sean-k-mooney | when i said recent i ment in the last month or so | |
| 14:32:08 | sean-k-mooney | nova force-delete does exactly the same thing | |
| 14:33:05 | opendevreview | sean mooney proposed openstack/osc-placement master: default to max version when no session https://review.opendev.org/c/openstack/osc-placement/+/794276 | |
| 14:33:24 | sean-k-mooney | gibi: filed https://storyboard.openstack.org/#!/story/2008958 and updated ^ | |
| 14:33:39 | sean-k-mooney | gibi: let me know if there is anything else i need to do | |
| 14:34:11 | sean-k-mooney | storyboard is still kind of a pain to naveigate but its got much faster then i rememeber | |
| 14:39:30 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: default to max version when no session https://review.opendev.org/c/openstack/osc-placement/+/794276 | |
| 14:40:11 | gibi | sean-k-mooney: thanks. just added the task number to the commit message too ^^ | |
| 14:40:25 | gibi | this will connect the review to the task in storyboard | |
| 14:40:28 | sean-k-mooney | oh ok i have seen both but i tought task was optional | |
| 14:40:30 | gibi | and update its state | |
| 14:40:33 | sean-k-mooney | oh ok | |
| 14:40:37 | sean-k-mooney | cool | |
| 14:40:56 | sean-k-mooney | am general question | |
| 14:41:13 | sean-k-mooney | if placment is going to remain a deliverable of compute | |
| 14:41:31 | sean-k-mooney | do we want to continue to maintain both storyboard and lauchpad | |
| 14:41:54 | sean-k-mooney | i also just realised the placement repariting spec should have been proposed in the placement repo | |
| 14:42:06 | sean-k-mooney | since that is where placment specs are ment to be tracked | |
| 14:42:35 | gibi | sean-k-mooney: I checked the placement repo but it does not have specs folder | |
| 14:42:35 | sean-k-mooney | https://docs.openstack.org/placement/latest/specs/index.html | |
| 14:42:51 | sean-k-mooney | its in the docs folder | |
| 14:42:55 | sean-k-mooney | so that it renders there | |
| 14:42:56 | gibi | /o\ | |
| 14:43:12 | sean-k-mooney | https://github.com/openstack/placement/tree/master/doc/source/specs | |
| 14:43:49 | sean-k-mooney | i kind of forgot this was there until i rememebred a few days ago | |
| 14:44:52 | gibi | OK, I will move the re-prenting spec to the placement repo :/ | |
| 14:45:04 | sean-k-mooney | well i was not going to ask you to do that | |
| 14:45:18 | sean-k-mooney | just going forward do we want to keep the split | |
| 14:45:39 | sean-k-mooney | e.g. nova has nova-specs and launchpad placement has in repo specs and sotry borad | |
| 14:46:03 | sean-k-mooney | this is more a wider team question or mailing list topic | |
| 14:46:17 | gibi | I don't feel the power to do a migration from storyboard | |
| 14:46:21 | sean-k-mooney | but that bug just reminded me of it | |
| 14:46:59 | gibi | we have very few incoming things in placemnet | |
| 14:47:42 | sean-k-mooney | true | |
| 14:47:55 | sean-k-mooney | do we want to add it to the meeting adgenda | |
| 14:48:08 | sean-k-mooney | we have the bug review for nova in lanuch pad | |
| 14:48:17 | sean-k-mooney | do we have somethign that will do the same for story borad | |
| 14:48:37 | sean-k-mooney | e.g. a linke to the open stories | |
| 14:48:58 | sean-k-mooney | its pretty stable and well tested so placment bugs are rare | |
| 14:50:55 | gibi | sean-k-mooney: please put it up to the agenda. Honestly I don't think it worth the effort to do extra things with placement but it make sense to discuss it | |
| 14:52:12 | sean-k-mooney | oh i was suggesting adding a "check for open placment bugs" item | |
| 14:52:19 | sean-k-mooney | like when we do the nova bug review | |
| 14:52:38 | sean-k-mooney | more just to prompt use to look for placment bugs if tehy areever reported | |
| 14:53:00 | sean-k-mooney | most get brought up here anyway since nova typicly is what finds them if they exist | |
| 14:57:13 | gibi | honestly I don't belive that people will start looking at open bug reports in placement by that prompt. but I might be negative | |
| 15:01:31 | sean-k-mooney | ok then lets leave it be so | |
| 15:03:43 | melwitt | tosky: guh, ok, I was wondering if there might be something like that since the fails. I'll look into a potential squash, thanks for the hint | |
| 15:04:00 | gibi | I just don't have the bandwidth to do a proper care for placement as well, but it seem that placement does not require that level of care most of the time | |
| 15:05:02 | masterpe[m] | Hi, we have Train running, yesterday we got the error "No valid host was found. There are not enough hosts available" on our platform. If I count the free RAM and vcpu of the nodes, I should have enough resources left to get the instances started. Is there something that I miss? | |
| 15:05:36 | melwitt | sean-k-mooney, gibi: I knew that placement had its own specs repo but I thought it was intentional that proposed spec to the nova-specs repo since the projects are back together again /shrug | |
| 15:09:53 | melwitt | gibi: there is a review I have hanging around if you might be able to spare some time, it is not urgent https://review.opendev.org/c/openstack/nova/+/780395 | |
| 15:15:38 | sean-k-mooney | melwitt: ya so i didnt know i we had decied to not use the in repos specs or not | |
| 15:15:54 | sean-k-mooney | e.g. if we had jsut defaulted to nova-spec or if it was intentional | |
| 15:16:31 | melwitt | sean-k-mooney: no, I'm not saying there was a decision, I had assumed something like that had occurred because of where the re-parent spec was proposed, but I was wrong | |
| 15:19:52 | gibi | melwitt: queued that up for review | |
| 15:20:24 | melwitt | thank you gibi | |
| 15:26:23 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix typos in minimum version policy docs https://review.opendev.org/c/openstack/nova/+/795575 | |
| 15:26:24 | melwitt | masterpe[m]: you need to look at the debug level logs in the placement api logs and the nova-scheduler logs to see details about why no valid host was found | |
| 15:28:39 | masterpe[m] | To solve it for now I enabled a node that was disabled/marked as spare. So I need to enable debug mode to troubleshoot? | |
| 15:32:42 | melwitt | masterpe[m]: to get full detail, yes. what you can do without debug mode is check whether there is a message like "no allocation candidates" in the nova-scheduler log, if you see that, it's likely cpu/ram/disk (as those resources are filtered by placement service) | |
| 15:33:42 | melwitt | if you don't see that, it means one or more of the nova side scheduler filters eliminated all hosts | |
| 15:34:11 | opendevreview | Balazs Gibizer proposed openstack/placement master: [DNM]: test consumer_type rollback due to consumer_gen conflict https://review.opendev.org/c/openstack/placement/+/795577 | |
| 15:36:31 | melwitt | gibi: oh heh, I had wondered if you saw my latest confusing comment. I was thinking the same thing this morning that it would be best to check what happens with a test. thanks for doing that | |
| 15:36:59 | gibi | melwitt: yeah, you are right the whole PUT /allocations/{uuid} call is not a single transaction | |
| 15:37:28 | gibi | so with the current implementation we can leak a consumer_type update from a rejected call :/ | |
| 15:39:31 | gibi | unfortunatly create a test case for a real race is too hard | |
| 15:39:43 | melwitt | gibi: ah, I see your new comment on the review. yeah ok, agree we need something higher up the stack then in order to not leak (or some other better idea yet to be thought of) :( I will work on it | |
| 15:40:32 | gibi | it is hard :/ | |
| 15:42:25 | melwitt | yeah, would require faking things with mocks at best. I didn't explore that yet so not sure how messy or possible | |
| 15:46:49 | spatel | sean-k-mooney periodically i am seeing this issue on my nova logs not sure if this is bug or not - http://paste.openstack.org/show/806496/ | |
| 15:51:45 | melwitt | spatel: that error means connection to the database failed while trying to write a service record update (heartbeat), not a bug. error being emitted from pymysql client | |
| 15:52:41 | spatel | melwitt today i said terraform to build 20 vm and 4 vm failed and stick in BUILD process, when i check their compute nodes i found that error | |
| 15:55:44 | melwitt | spatel: that paste wouldn't be directly related to a build failure (that paste is a service record update for a service heartbeat) so you'd need to look at which errors are related to the build fail, if any | |
| 16:03:13 | spatel | let me see, may be issue of rabbitmq in that case | |